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FOREWORD 


This  report  presents  behavioral  observations  and  human 
performance  analyses  obtained  by  the  U,S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  in 
support  of  a  concept  exploration  conducted  by  the  Institute 
for  Defense  Analyses  (IDA)  Simulation  Center.  As  a  part  of 
the  1996  Defense  Science  Board  Summer  Study  "Tactics  and 
Technology  for  21st  Century  Military  Superiority,"  IDA 
sponsored  a  set  of  excursions  into  the  21st  Century  virtual 
battlefield.  ARI  provided  technical  advisory  service  (TAS) . 
This  assistance,  related  to  but  not  a  part  of  the  ARI  work 
program,  is  an  important  part  of  the  Army  and  Department  of 
Defense' s  training  and  personnel  performance  research  and 
development  mission. 

Personnel  from  three  ARI  research  units  (the  Infantry 
Forces  Research  Unit  at  Fort  Banning,  the  Simulator  Systems 
Research  Unit  at  Orlando,  and  the  Advanced  Training  Methods 
Research  Unit  in  Alexandria)  worked  together  with  IDA 
personnel  on  the  virtual  simulation-based  excursion  entitled 
"The  Small  Team  Portal  Into  the  21st  Century  (STP21) ."  Data 
collected  from  the  virtual  environment's  conceptual 
battlefield  provided  insight  into  future  small  team 
capabilities  and  demonstrated  the  value  of  virtual 
simulation  environments  for  concept  testing. 


ZITA  M.  SIMUTIS  EDGAR  M.  JOHNSON 

Technical  Director  Director 
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SMALL  TEAM  PORTAL  INTO  THE  21ST  CENTURY— SPT21 


EXECUTIVE  SUMMARY 


Research  Requirement: 

As  a  part  of  the  1996  Defense  Science  Board  (DSB) 

Summer  Study  "Tactics  and  Technology  for  21st  Century 
Military  Superiority,"  the  Institute  for  Defense  Analyses 
Simulation  Center,  assisted  by  behavioral  scientists  from 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  conducted  a  set  of  excursions  designed  to 
provide  analytical  insights  about  concepts  and  technologies 
being  considered  for  small  team  operations  on  the  DSB' s 
conceptual  21st  Century  battlefield. 

The  DSB  focus  was  on  the  concept  of  using  technology  to 
enable  small,  rapidly  deployable  forces  to  accomplish 
missions  previously  only  available  to  larger  forces.  An 
additional  area  of  interest  was  the  viability  of  a  virtual 
simulation  for  concept  exploration. 

Procedure: 

This  report  documents  a  series  of  exercises  conducted 
in  a  virtual  simulation  environment  using  man-in- the-loop 
virtual  excursions  into  the  21st  Century  battlefield.  U.S. 
Army  and  Marine  Corps  personnel  used  specially  designed 
devices  in  a  virtual  simulation  facility  to- test  concepts 
about  the  capabilities  of  small  (3-  to  12-man)  teams 
operating  in  a  sensor  rich  environment.  In  addition  to 
computer  recorded  performance  data,  behavioral  and  tactical 
observers  documented  soldier  performance  and  interactions 
with  specific  equipment. 

Findings : 

Combat  effectiveness,  although  hampered  by  constraints 
from  present  day  and  prototype  equipment,  was  enhanced 
through  sophisticated  communication  devices  and  computers. 
Personnel  were  able  in  the  virtual  environment  to  perform 
tasks  similar  to  those  that  might  occur  in  a  future 
battlefield  scenario.  A  benefit  of  the  simulation  was  the 
ability  to  portray  future  missions  with  prototype  equipment 
to  determine  potential  for  success.  An  additional  benefit 
came  from  the  ability  to  provide  man-in-the-loop  human 
performance  data  in  the  initial  stages  of  concept 
exploration. 


vii 


utilization  of  Findings: 

The  results  of  these  excursions  were  briefed  back  to 
the  DSB  in  August  1966.  Intermediate  briefings  and 
demonstrations  in  the  virtual  simulation  facility  were 
provided  as  in~progress  reviews  and  to  sponsors  throughout 
the  course  of  the  research.  It  is  apparent  that  the 
potential  for  future  experimentation  and  concept  exploration 
using  the  techniques  of  virtual  simulation  offers  many 
benefits. 
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Small  Team  Portal  Into  the  21st  Century— STP21 


Introduction 

In  response  to  a  request  from  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology,  the  Defense  Science  Board  (DSB) 
initiated  a  1996  Summer  Study  "Tactics  and  Technology  for  21st 
Century  Military  Superiority."  The  overall  DSB  focus  was  on  the 
concept  of  enabling  relatively  small  rapidly  deployable  forces  to 
accomplish  missions  heretofore  only  possible  with  much  larger  and 
massed  forces.  Small  (3-12  man)  teams,  specially  equipped  and 
trained,  would  achieve  enhanced  combat  capability  from  remote 
sensors  and  weapons.  Questions  revolved  around  mission 
expectations,  operational  and  technical  capabilities  to  enhance 
team  effectiveness,  flexibility,  and  survivability. 

The  DSB  requested  assistance  from,  among  others,  the 
Institute  for  Defense  Analyses  (IDA)  Simulation  Center  in 
Alexandria,  VA.  IDA  was  to  provide  analytical  insights  about 
concepts  and  technologies  being  considered  for  small  team 
operations  on  the  DSB's  conceptual  21st  Century  Battlefield 
(2010-2015  timeframe)  by  developing  a  virtual  simulation  system 
to  support  man-in-the-loop  excursions.  The  system  had  a 
simulated  intermediate  leader  portal,  two  small  team  portals,  and 
control  cells  representing  higher  headquarters,  support  elements, 
and  computer  generated  forces. 

Excursion  Concept 


The  study  of  rapidly  deployable,  specially  equipped  teams 
supported  by  remote  sensors  has  several  target  audiences  and 
proponents.  The  IDA  simulation  was  named  Small  Team  Portal  into 
the  21st  Century  (Virtual)  Battlefield,  or  STP21.  Related 
programs  include  the  U.S.  Marine  Corps  Sea  Dragon,  the  Training 
and  Doctrine  Command  (TRADOC)  Army  After  Next,  the  U.S.  Army 
Infantry  School  Rapid  Force  Projection  Initiative  (RFPI) ,  and  the 
Defense  Advanced  Research  Projects  Agency  (DARPA)  Small  Unit 
Operations  (SUO) . 

IDA'S  goal  was  to  assess  small  unit  battlefield  operations, 
and  at  the  same  time,  to  evaluate  the  STP21  virtual  simulation  as 
a  tool  for  studying  future  operations.  More  specifically,  the 
STP21  exercise  was  designed  to  investigate  the  ability  of  small 
independently  operating  units  to  remotely  place  precision  fires 
on  long  range  targets,  using  sophisticated  sensor  suites  and 
digitized  equipment.  In  addition,  the  intent  was  to  assess  the 
ability  of  small  groups  of  soldiers,  in  a  semi-autonomous  mode, 
to  perform  functions  formerly  done  by  a  larger  force  with  more 
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equipment.  (See  Coe,  Madden,  Mengel,  &  Wright,  1996,  for  greater 
detail . ) 

Data  collection  addressed  three  essential  elements  of 
analysis  (EEA) : 

EEA  1  explored  enhancement  of  small  team  situational 
awareness  through  sensors  and  two  computer  devices .  palmtop 
personal  data  assistants  (PDA) ,  and  the  laptop  Map  of  the  Future 
(MOF)  . 


EEA  2  explored  enhancement  of  combat  effectiveness  by 
measuring  remote  fires  and  use  of  specialized  devices. 

EEA  3  examined  the  utility  of  individual  and  small  team 
virtual  simulation  in  concept  exploration. 

Army  Research  Institute  Role 

IDA  requested  that  behavioral  scientists  from  the  Army 
Research  Institute  (ARI)  participate  in  the  simulation  exercise 
as  observers  during  the  tactical  scenarios.  ARI  was  to 
3QYeral  broad  areas.  Issues  included  the  small  team^ s  ability  to 
manage  large  amounts  of  data,  training  on  surrogate  equipment, 
man-machine  interface,  and  identification  of  the  slcills  needed  to 
operate  on  the  DSB' s  conceptual  21st  Century  Battlefield. 

Five  research  psychologists  from  ARI  (two  from  the  Infantry 
Forces  Research  Unit  at  Fort  Banning,  GA,  two  from  the  Simulator 
Systems  Research  Unit  at  Orlando,  FL,  and  one  from  the  Advanced^ 
Training  Methods  Research  Unit  at  Alexandria,  VA)  participated  in 
the  evaluation  from  June  through  August,  1996.  ARI  developed  and 
administered  questionnaires  to  measure  participant  attitudes 
toward  and  difficulties  with  use  of  the  specialized  equipment, 
conducted  interviews,  and  observed  the  scenarios. 

This  report  documents  the  observations  made  and  research 
performed  in  support  of  the  IDA  DSB  concept  explorations.  It 
comments  on  four  specific  areas  related  to  the  concept  of  small 
unit  operations:  equipment,  personnel,  training,  and  simulation. 

ARI  Perspective  on  the  Value  of  Simulation 

Virtual  simulations  can  enhance  the  concept  development 
process  by  providing  an  opportunity  to  assess  mission  performance 
with  humans  "in  the  loop,"  performing  tasks  as  they  are  expected 
to  once  the  system  is  fielded.  Information  about  their 
performance  can  be  collected  and  analyzed  to  evaluate  the  concept 
and  identify  ways  to  improve  it.  The  information  can  vary  from 
empirical  data  about  response  times  and  performance  accuracy  to 
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more  subjective  reports  by  subject  matter  experts  about  problems 
likely  to  be  encountered  in  a  field  situation.  Virtual 
simulations  permit  this  information  to  be  obtained  prior  to  the 
development  of  a  complete  physical  prototype. 

Information  about  how  humans  perform  in  these  virtual 
simulations  can  be  used  in  several  ways.  First,  it  can  help  to 
assess  the  overall  feasibility  of  the  concept  itself.  (For 
example,  can  human  operators  detect  targets  and  submit  fire 
requests  rapidly  enough  to  accomplish  the  mission  successfully?) 
Second,  it  can  help  to  refine  the  system  functional  requirements. 
(Elevated  sensor  magnification  must  be  at  least  6X  or  targets 
cannot  be  detected  beyond  2  km.)  Third,  it  can  help  design  tasks 
and  organizations,  assign  system  functions  to  human  operators  or 
automated  components,  and  organize  human  functions  into  duty 
positions.  (Each  element  requires  three  personnel  for  effective 
operations.)  Fourth,  it  can  help  identify  the  necessary 
abilities  and  prerequisite  skills  required  to  perform  the  jobs 
successfully.  (All  operators  require  a  high  degree  of  general 
computer  literacy  and  understanding  of  friendly  and  enemy 
battalion-level  tactics.)  Finally,  it  can  identify  training 
needs  and  support  the  development  and  evaluation  of  operator 
training. 


Approach  and  Procedures 
Simulation  Center  Facility 


The  IDA  Simulation  Center,  conceptualized  as  an  integrating 
center,  permits  a  variety  of  excursions  and  explorations. 
Commercial  off-the-shelf  equipment  and  prototype  specialty 
equipment  are  integrated  as  needed  for  concept  demonstrations  and 
experimentation.  The  facility  is  permanent,  but  on-site 
equipment  varies  with  the  specific  area  of  investigation,  thus 
the  concept  of  an  integrating,  rather  than  dedicated  facility. 

Participants 

Two  U.S.  Marine  Corps  captains  served  as  Fire  Direction 
Center  (FDC)  controllers.  During  the  first  week  of  testing,  two 
Army  National  Guard  captains,  supplemented  by  IDA  civilian 
personnel,  participated  as  test  subjects  for  early  operational 
trials.  In  the  second  week,  six  Marine  second  lieutenants 
participated  in  various  team  configurations.  For  example,  in  one 
configuration,  two  were  on  one  team,  three  on  another,  and  one 
served  as  the  intermediate  team  leader. 
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General  Concept 


Two  weeks  of  data  collection  were  planned.  According  to  the 
oj^iginal  plan^  an  overview  and  training  on  the  eguipment  would  be 
followed  by  coin.puter~based  simulation  exercises  during  the 
remainder  of  the  week.  Each  day^s  trials  would  end  with  after 
action  reviews  (AARs)  to  elicit  lessons  learned.  The  second  week 
was  to  repeat  the  first  with  different  test  subjects.  Each  day 
was  to  be  increasingly  more  complex,  to  test  the  limits  of  the 
simulation  and  to  develop  engagement  timelines  to  measure 
situational  awareness  and  combat  effectiveness.  The  opposing 
force  (OPFOR) ,  terrain,  and  sensors  were  to  be  varied,  as  well  as 
team  size  and  mobility.  ARI  was  to  address  the  human  factors  and 
behavioral  interface  issues.  Specific  items  of  interest  were 
operator  workload,  equipment  problems  that  impacted  on  test 
subject  ability  to  conduct  the  mission,  and  simulator  sickness  or 
fatigue . 

Difficulties  interconnecting  the  prototype  equipment  forced 
changes  to  data  collection  and  the  overall  plan.  The  first  week 
became  a  pilot  week  to  resolve  problems  and  make  equipment 
changes.  Standing  operating  procedures  (SOPs)  for  equipment  use 
and  radio  communications  were  developed.  The  second  week 
provided  data  and  performance  measures  related  to  the  EEAs . 

Facilities  and  Equipment 

The  Simulation  Center  consisted  of  an  exercise  control  area 
(EGA),  two  small  team  portals,  and  an  intermediate  team  leader 
portal/station,  plus  tactical  (military)  and  behavioral  observer 
stations.  The  small  team  and  leader  stations  were  networked  to 
the  EGA  that  supported  the  team  portals.  (See  layout  at  Figures 
1  and  2.)  EGA  equipment  included  the  simulated  FDC/Fire  Support 
Station,  the  OPFOR  Control  Center,  and  the  Remote  Sensor 
Center/Intelligence  Station.  A  briefing  area  was  available. 


Figure  i.  STP21  Simulation  Center  concept  exploration  facility. 
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Figure  2.  STP21  simulation  facility  layout. 

The  Fire  Support  Station  received  and  responded  to  calls 
from  the  team  portals.  The  remote  fire  station  delivered 
artillery,  missile,  naval  gunfire,  and  close  air  support  in 
response  to  preplanned  fire  requests  or  requests  for  fire  from  a 
team  or  intermediate  leader.  After  player  calls  for  fire, 
weapons  were  selected  and  status  reported.  This  station  was 
operated  by  the  Marine  captains,  assisted  by  an  IDA  civilian. 

IDA  also  staffed  the  Sensor  Station  and  the  Modular  Semi- 
Automated  Forces  (ModSAF)  OPFOR  control  center. 

The  intermediate  leader  station  was  a  coordination  cell 
located  in  an  area  adjacent  to  one  of  the  teams.  Teams  had  voice 
radio  communication  with  the  leader,  and  with  each  other,  as  well 
as  with  the  FDC.  Each  player  had  earphones  for  aural  replication 
of  battlefield  noises  and  engine  sounds  in  addition  to  normal 
radio  traffic;  only  one  key  player  at  each  station  had  radio 
communication.  The  team  portals  and  the  intermediate  leader 
station  equipment  could  be  observed  by  the  operational  observers 
and  behavioral  analysts.  Observer  monitors  slaved  to  the  team 
equipment  replicated  the  soldiers'  views^  the  virtual  terrain  was 
easily  visible  from  all  locations. 

The  team  stations  had  both  disparate  and  common  elements  of 
equipment.  Portal  1  (Figure  3),  was  a  two  person  station  with  a 
treadmill/treadport  motion  platform  device.  The  motion  of  the 
treadport  simulated  the  soldier  walking  over  terrain.  The 
terrain,  a  digitized  data  base,  was  depicted  in  the  WISE  (Walk-In 
Synthetic  Environment) .  The  WISE  had  three  large  screens  which, 
via  a  rear  projection  device,  depicted  the  terrain  and  thus 
appeared  to  surround  the  soldier.  The  soldier  on  the  treadport 
could  stand  still,  walk  or  run.  (The  device  would  not  support  a 
view  of  the  battlefield  from  the  crawl  position.)  A  surrogate 
vehicle  was  also  available  at  this  portal. 
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Treadoort  Motion  Platform 


Figure  3.  Portal  1. 


Portal  2  (Figure  4)  was  a  two-person,  then  eventually  a 
three-person  "mobile"  station,  where  one  person  could  drive,  in 
addition  to  assisting  in  target  location.  The  vehic  e, 

the  Dial-A-Tank,  had  a  steering  yoke  shift  lever,  and  foot  peda 
which  the  operator  could  use  to  control  forward  and  reverse 
motion,  acceleration  and  braking,  and  direction.  The  terrain 
data  base  was  depicted  on  two  large  screens  to  the  front  of  the 
vehicle.  In  actuality  the  team  was  seated  at  a  table. 


Figure  4.  Portal  2. 

The  Melios  (Mini  Eyesafe  Laser  Infrared  Observation  Set)  was 
the  sensor  device  used  at  Portal  1-  A  variation  on  current 
equipment,  Melios  was  a  ten  power  binocular-like  hand  held 
device.  It  had  an  embedded  laser  range  finder  that  permitted  tne 
operator  to  laze  on  a  target,  with  target  range  and  azimuth 
depicted  on  a  small  screen  display  monitor.  Target  information 
was  relayed  back  to  the  FDC.  The  system  featured  a  second  laze 
capability  designed  to  predict  the  location  of  the  target  over 
time . 
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The  sensor  used  at  Portal  2  was  known  as  COVER.  A  simulated 
tethered  aerial  sensor,  COVER  could  be  elevated  from  tree  top 
level  up  to  300  feet  in  the  air  above  the  team's  position,  and 
turned  for  a  360  de^^s®  view.  The  COVER,  like  the  Melios,  had  a 
laser  range  finder  that  gave  range  and  azimuth  to  any  target 
detected.  It  was  designed  to  provide  both  optical  and  thermal 
views,  although  only  the  day  capability  was  demonstrated.  COVER 
was  operated  by  a  joystick;  the  input  and  view  from  the  sensor 
were  depicted  on  a  small  screen  monitor  at  the  soldier's  station. 

Each  portal  had  two  data  entry/receipt  devices  for  text  and 
graphic  data  communication  and  display.  The  Grunt  was  linked 
either  to  the  Melios  or  the  COVER  device.  Grunt  was  a  small 
computer  keypad  with  data  entry  by  means  of  a  stylus.  It 
displayed  the  range  and  azimuth  from  the  Melios  or  COVER, 
together  with  global  positioning  system  (GPS)  information,  and 
enabled  an  operator  to  call  for  fire.  Preformatted  messages 
could  be  accessed  and  sent  after  the  operator  inserted  grid 
location  and  other  selected  information.  Free  text  messages 
could  be  sent,  but  Grunt  had  no  receiving  capability. 

The  other  data  entry  device  was  the  Map  of  the  Future  (MOF) . 
The  MOF,  conceptually  a  pocket-sized  fold  up  electronic  map  but 
represented  in  the  simulation  by  a  laptop  computer,  displayed  an 
electronic  digital  map.  The  map  was  the  same  terrain  database 
depicted  in  the  WISE  at  Portal  1  and  on  the  wide  screen  at  Portal 
2.  Any  section  of  a  standard  1:50,000  map  sheet  could  be  viewed, 
and  expanded  or  minimized  through  a  zoom  feature.  The  MOF  screen 
presented  a  small  top-down  view  of  the  entire  database  and  a 
larger  display  of  a  map  section  of  a  particular  slice  of  terrain. 
The  smaller  display  contained  a  "pan"  box  that  the  user  could 
move  to  determine  which  map  section  was  displayed.  Friendly  and 
enemy  positions,  man-placed  sensors,  remotely  monitored 
battlefield  sensor  systems  (REMBASS) ,  and  minefields  could  also 
be  positioned  and  represented  on  the  map.  Messages  appeared  on 
the  MOF  as  E-mail,  and  calls  for  fire  could  be  sent  through  E- 
mail  back  to  the  FDC.  The  MOF  message  function  was  used  for 
sending  and  receiving  sensor  and  target  information,  and  for 
updates  on  target  status. 

Excursion  Scenarios 


In  the  general  scenario,  the  U.S.  supported  a  European 
nation.  After  the  threat  was  turned  back,  U.S.  personnel  were 
inserted  from  a  precision  strike  battalion  to  maintain  regained 
ground.  This  follow-on  force,  the  small  teams,  was  positioned 
and  equipped  with  devices  for  detecting  and  identifying  targets, 
processing  fire  requests,  and  communicating  these  requests  to  a 
fire  direction  center  at  another  location. 
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The  overall  intent  was  to  specify  targets  and  their  priority 
for  destruction,  using  the  varying  sensor  capabilities  available 
to  the  U.S.  team.  Measures  of  effectiveness  included  target 
detection,  identification  and  request  time,  numbers  of  targets 
attacked,  area  of  control,  workload  and  task  management. 

The  players  were  given  overall  rules  of  engagement  with 
instruction  on  positive  identification  of  targets  to  preclude 
civilian  casualties.  Players  were  to  observe  the  battlefield, 
call  for  and  adjust  fires  on  valid  targets,  and  designate  targets 
as  appropriate.  They  were  to  report  battle  damage,  coordinating 
with  each  other  and  the  intermediate  leader.  The  intermediate 
leader's  role  was  to  exercise  command  and  control  over  the 
targeting  element,  to  plan  the  operation,  and  to  service  targets 
if  appropriate.  The  focus  was  on  the  small  team  portals. 

The  primary  terrain  data  base  was  the  wooded  mixed  terrain 
of  Hohenfels,  Germany,  shown  in  the  Synthetic  Theater  of  War, 
Europe  (STOW-E)  database.  Some  excursions  employed  the  Hunter- 
Ligget  desert  database  of  29  Palms,  California,  or  the  simulated 
urban  terrain  of  the  McKenna  MOUT  Site  at  Fort  Banning,  Georgia. 
All  exercises  were  in  daylight  mode.  Potential  threat  targets 
were  tank  and/or  BMP  platoons,  trucks,  and  dismounted  infantry. 

Data  Collection 


Automated  data  collection  provided  by  the  computer  system 
provided  information  on  target  presentation,  exposure  times,  time 
to  hit  and  kill,  calls  for  fire,  and  battle  damage  assessment. 
Data  were  collected  for  both  portals,  and  for  all  sensor  suites 
and  weapon  systems.  These  data,  transparent  to  the  players,  were 
collected  by  IDA's  distributed  interactive  simulation  network  to 
supplement  the  observational  data  collected  by  tactical  and 
behavioral  observers. 

The  operational  and  behavioral  observers  maintained 
worksheets  (Appendix  A)  to  record  elapsed  times  and  significant 
events  during  the  excursions.  Verbatim  comments  from  the  players 
were  also  noted,  as  well  as  the  observers'  informal 
interpretations  of  what  was  happening.  Observer  data  sheets  were 
collected  daily,  and  their  content  linked  to  the  computer 
generated  data.  Video  coverage  was  provided  by  a  film  crew  that 
was  making  a  demonstration  tape  to  document  the  simulation 
exercises  and  some  after  action  reviews  for  DSB  personnel. 

Questionnaires 


Several  questionnaires  were  developed  to  elicit  information 
from  the  participants  before  and  after  the  excursions.  The  STP21 
Questionnaire  (Appendix  B)  obtained  background  data  on  previous 
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military  experience  and  training,  familiarity  with  items  of 
equipment,  and  their  ability  to  focus  attention  on  various 
activities.  (Focus  items  came  from  the  Immersive  Tendencies 
Questionnaire,  Witmer  and  Singer,  1994.)  This  questionnaire  was 
administered  on  the  first  day,  prior  to  the  first  excursion. 

The  Health  Questionnaire  (Appendix  C)  obtained  information 
on  the  participants'  current  state  of  health.  It  was  administered 
on  the  first  day  of  the  excursions,  after  initial  training  but 
prior  to  the  first  excursion,  and  on  each  subsequent  day  prior  to 
the  start  of  the  excursions. 

A  potential  problem  with  application  of  simulation 
technology  in  the  context  of  military  concept  development  is  that 
participants  may  experience  side  and  aftereffects  similar  to,  but 
not  limited  to,  symptoms  of  motion  sickness.  Simulator  sickness 
is  a  concern  because  it  can  potentially  distract  or  in  other  ways 
degrade  the  performance  of  the  participants.  The  Simulator 
Sickness  Questionnaire  (SSQ)  (Appendix  D)  was  administered  prior 
to  the  first  excursion  and  after  the  last  excursion  each  day 
(before  and  after  immersion)  to  take  pre-existing  symptoms  into 
account.  (See  Lampton,  Kraemer,  Kolasinski  &  Knerr  (1995)  and 
Kennedy,  Hettinger,  &  Lilienthal,  (1988)  for  further 
information . ) 

The  Human  Factors  Equipment  and  Functions  Questionnaire 
(Appendix  E)  determined  the  frequency  with  which  participants 
performed  various  tasks  and  functions  and  the  degree  of 
difficulty  they  had  performing  those  tasks.  It  was  administered 
at  the  end  of  each  day.  The  Learning  Questionnaire  (Appendix  F) 
was  used  to  determine  difficulty  participants  had  learning  to 
perform  various  tasks.  It  also  was  administered  at  the  end  of 
each  day. 

The  Simulator  Questionnaire  (Appendix  G)  asked  the  extent  to 
which  the  participants  felt  immersed  in  the  simulation.  The 
items  were  taken  from  a  Presence  Questionnaire  developed  by 
Witmer  and  Singer  (1994) .  The  Simulator  Questionnaire  was 
administered  after  the  last  excursion  on  the  second  and  fourth 
days  of  week  two. 

After  Action  Reviews  and  Interviews 


After  Action  Reviews  (AARs) .  At  the  conclusion  of  each 
excursion,  the  Exercise  Controller  conducted  an  AAR.  The  purpose 
of  the  AAR  was  to  solicit  comments,  lessons  learned,  and  other 
observations.  While  each  AAR  was  attended  by  the  scenario  role 
players,  observers,  and  some  technical  support  personnel  in 
addition  to  the  exercise  participants  themselves,  the  primary 
focus  was  on  obtaining  information  from  the  participants.  Each 
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AAR  had  two  segments.  The  first  focused  on  the  tactical  events, 
the  second  on  aspects  of  the  simulation. 

Group  interview.  The  ARI  researchers  led  a  group  interview 
on  the  morning  of  the  last  day  of  the  second  week.  The  intent 
was  to  solicit  information  from  the  participants  about  manpower, 
personnel,  and  training  issues.  Questions  asked  were: 

•  What  equipment  or  procedures  worked  best?  What 
worked  least  well? 

•  What  was  the  hardest  task  for  you  to  perform?  What 
was  the  easiest? 

•  What  equipment  or  procedures,  if  any,  did  you  need 
more  training  on? 

•  What  equipment  or  procedures  did  you  have  the  most 
confidence  in?  What  equipment  or  procedures  did  you 
have  the  least  confidence  in? 

•  What  distracted  you  the  most? 

•  If  you  participated  in  the  same  exercises  again, 
would  you  do  anything  differently?  If  so,  what? 

•  What  prior  experience  or  training  (if  any)  helped 
you  in  these  exercises? 

•  How  many  people  should  be  in  an  observer  group? 

Why? 

•  What  was  the  most  difficult  duty  position?  What  was 
the  easiest?  Why? 

•  Do  you  have  any  ideas  about  how  to  share  the 
workload? 

•  What  do  you  think  a  man  can  do  here  that  a  computer 
can' t? 

•  What  did  you  have  to  do  that  you  think  should  be 
done  by  computer  instead  of  a  human? 
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Results 


Participants 

Serving  in  the  portals  were  six  male  Marine  Corps  second 
lieutenants.  All  were  in  the  active  component,  and  were  in 
student  status.  All  had  BA  or  BS  degrees.  In  terms  of 
experience,  there  were  two  distinct  groups  of  three  each.  The 
members  of  the  first  group  had  recently  completed  Officer 
Candidate  School  (OCS)  and  had  a  mean  of  seven  months  of  milit  y 
service.  The  members  of  the  second  group,  who  were  awaiting  the 
start  of  OCS,  all  had  prior  enlisted  service  (mean  -  7  years,  iu 
months) .  Four  of  the  six  had  had  some  forward  observer  training. 
None  were  experienced  in  navigating  in  simulated  terrain 
databases.  None  had  been  to  Desert  Shield/Desert  Storm  or  to  a 
Combat  Training  Center. 

Their  experience  with  the  technology  used  or  simulated  in 
these  excursions  is  shown  in  Table  1.  The  generally  high  level 
of  experience  with  laptop  computers,  and  to  a  lesser  extent, 
sending  and  receiving  E— mail,  is  worthy  of  note. 


Table  1 

Participant’  Experience  with  Relevant  Technology 


Experience 

Technology 

None 

Limited 

Moderate 

Considerable 

Global  Positioning 
System  (GPS) 

2 

3 

1 

0 

Laser  range  finders 

5 

1 

0 

0 

Target  designators 

5 

1 

0 

0 

Helmet  mounted 
displays 

5 

1 

0 

0 

SIMNET-type 

environments 

6 

0 

'  0 

0 

JANUS 

6  i 

0  n 

0 

0 

Sending/receiving  E- 
mail 

2 

1 

2 

1 

Laptop  computers 

0 

2 

2 

2 

Thermal  sights 

3 

3 

0 

_ _  ^  ^ 

0 

/-it 

JL  li  J..  - - 

reporting  that  level  of  experience, 
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Participant  Immersive  Tendencies 


Because  of  the  distractions  inherent  in  the  simulation 
facility,  participants  were  asked  about  their  ability  to  become 
involved  in  activities  and  to  ignore  distractions.  They  were 
also  asked  about  experience  with  computer  and  video  games. 
(Responses  are  shown  in  Appendix  H.)  The  group  showed 
considerable  variability  in  responses  to  the  questions  that  asked 
about  involvement  in  video  games,  sports,  and  other  activities. 

However,  most  reported  that  they  were  good  at  blocking  out 
distractions,  concentrated  well  on  enjoyable  activities,  switched 
attention  among  activities  well,  and  were  not  easily  disturbe 
when  working  on  a  task.  While  only  one  reported  more  than 
occasional  playing  of  arcade  or  video  games,  three  reported  more 
than  occasional  playing  of  computer  games. 

Equipment  Use 

The  Human  Factors  Equipment  and  Functions  Questionnaire 
provided  information  about  task  frequency  and  difficulty.  The 
Learning  Difficulty  Questionnaire  showed  how  hard  it  was  to  learn 
to  perform  certain  tasks.  While  most  tasks  were  equipment 
specific,  a  few  were  general.  Some  tasks  could  also  be  performed 
with  multiple  items  of  equipment.  Participants  completed  the 
questionnaires,  responding  only  for  pieces  of  equipment  that  they 
had  used  in  the  immediately  preceding  excursion.  Equipment  usage 
varied  considerably,  and  therefore  the  number  of  responses  on 
which  these  ratings  are  based  does  as  well.  For  example,  the 
Treadport  was  used  a  total  of  four  times  by  three  different 
participants,  while  the  Map  of  the  Future  was  used  a  total  of  24 
times  by  six  different  participants.  Questionnaire  means  were 
calculated  by  first  obtaining  the  mean  response  for  each 
participant  across  all  six  administrations  of  the  questionnaires, 
and  then  obtaining  the  mean  across  all  participants.  This  had 
the  effect  of  weighting  each  participant's  response  equally, 
regardless  of  the  number  of  times  each  used  a  particular  item  of 
equipment . 

COVER 

For  COVER  (see  Table  2),  distinguishing  friendly  and  neutral 
targets  was  rated  the  most  difficult  task.  As  will  be  presented 
later,  it  was  also  the  most  difficult  for  the  Melios,  the  other 
piece  of  equipment  that  required  direct  visual  observation  to 
make  this  determination.  However,  this  reflects  not  the 
equipment,  but  the  simulation,  which  presented  no  neutral 
entities  and  therefore  did  not  test  the  capability. 
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Table  2. 


Ratings  of  COVER  Tasks  (Means) 


Task 

Difficulty 

Frequency 

Learning 

Difficulty 

Distinguish  friendly  and 
neutral 

3.70 

2.90 

NA 

Making  joystick  work 
right 

3.33 

5.00 

NA 

Designate  targets 

3.00 

3.53 

2.13 

Identify  targets  (name 
them) 

2.97 

3.70 

2.63 

Using  the  joystick 
controller 

2.93 

4.83 

3.20 

Lazing 

2.63 

4.37 

2.13 

Detect  targets 

2.50 

4.10 

2.77 

Maintaining  orientation 

2.50  n 

4.70 

2.10 

Prioritize  targets 

2.33 

2.67 

NA 

Perform  BDA* 

2.20 

2.67 

2.30 

N  =  9  *Battle  damage  assessment 


Frequency  Key;  1-Never;  2-Occasionally;  3-Moderately  Often;  4- 
Frequently;  5-Almost  Constantly 

Difficulty  and  Learning  Difficulty  Key:  1-Very  Easy;  2-Easy;  3- 
Neither  Easy  nor  Difficult;  4-Difficult;  5-Very  Difficult 

Making  the  joystick  work  right,  designating  targets  and 
using  and  learning  to  use  the  joystick  controller  were  also 
difficult.  Designating  a  target  required  placing  a  cross-hair  on 
the  center  of  the  COVER  display  over  a  (sometimes  moving)  target 
using  the  joystick  controller.  The  joystick  had  some  undesirable 
handling  characteristics:  a  wide  area  of  insensitivity  around  the 
neutral  position,  with  high  sensitivity  beyond  that  band  made  it 
easy  to  overshoot  the  intended  destination. 

In  the  group  interview,  observing  a  single  target  with  the 
COVER  was  identified  as  one  of  the  easiest  individual  tasks  while 
tracking  multiple  targets  was  one  of  the  hardest.  The  viewing 
conditions  of  the  COVER  and  the  joystick  appeared  to  be  the 
cause.  Viewing  with  the  COVER  could  be  done  at  either  normal 
resolution  or  6X  magnification.  Our  crude  measurements  indicated 
that  the  horizontal  field  of  view  (FOV)  of  COVER  was  47  degrees 
at  normal  magnification,  and  4  degrees  at  6X.  The  resolution  of 
the  monitor  made  it  difficult  to  detect  distant  targets,  so 
participants  did  most  of  their  searching  and  tracking  at  6X. 

This  was  somewhat  similar  to  viewing  the  world  through  a  soda 
straw,  and  made  it  difficult  to  switch  back  and  forth  between  two 
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or  inor©  targets,  even  if  azimuth  and  range  were  known.  They  did 
not  slew  to  the  approximate  target  location  under  normal 
magnification,  then  increase  the  magnification;  they  later 
expressed  a  desire  for  a  zoom  capability  on  the  COVER. 

Melios 

For  Melios  (see  Table  3),  as  with  the  COVER,  and  as 
discussed  above,  the  most  difficult  task  was  distinguishing 
friendly  and  neutrals.  Seven  tasks  were  performed  with  both  the 
COVER  and  the  Melios.  Five  were  rated  as  more  difficult  on  the 
Melios,  and  two  rated  more  difficult  with  the  COVER. 

Table  3. 

Ratings  of  Melios  Tasks  (Means) 


Task 

Difficulty 

Frequency 

Learning 

Difficulty 

Distinguish  friendly  and 
neutral 

4.50 

2.75 

NA 

Designate  targets 

3.67 

3.00 

2.67 

Lazing 

3.25 

4.00 

NA 

Perform  EDA 

3.00 

2.75 

2.00 

Detect  targets 

2.75 

4.00 

2.08 

Identify  targets  (name 
them) 

2.50 

3.75 

2.50 

Prioritize  targets 

2.00 

3.00 

NA 

N  =  5 


Frequency  Key;  1-Never;  2-Occasionally;  3-Moderately  Often;  4- 
Frequently;  5-Almost  Constantly 

Difficulty  and  Learning  Difficulty  Key:  1-Very  Easy;  2-Easy;  3- 
Neither  Easy  nor  Difficult;  4-Difficult;  5-Very  Difficult 


This  may  represent  chance  variation,  but  it  may  also  reflect 
the  following.  COVER  was  employed  while  the  participant  was 
seated  in  a  chair  at  a  desk  but  the  Melios  was  employed  while  the 
participant  was  walking,  standing  or  sitting  on  the  Treadport, 
and  was  frequently  devoting  some  portion  of  his  attention  to 
maintaining  or  changing  his  position.  This  may  have  made  the 
Melios  tasks  appear  more  difficult.  Second,  the  Melios  is  a 
fairly  heavy  hand-held  device  (about  6  pounds)  and  holding  it  at 
eye  level  for  an  extended  period  caused  fatigue  and  instability; 
at  Portal  2  the  participant' s  elbows  could  be  braced  on  the  table 
for  support. 
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These  issues  were  raised  in  the  group  interview,  with 
disagreement  among  the  participants  on  whether  the  COVER  or  the 
Melios  was  better.  COVER  provided  a  more  stable  viewing  platform 
and  was  easier  to  use  with  eyeglasses.  The  use  of  Melios  was 
more  intuitive  than  COVER.  The  physical  movements  made  with 
Melios  made  it  easier  to  return  to  a  previous  position.  In  this 
sense,  Melios  contributed  to  better  situational  awareness.  The 
weight  of  Melios  was  criticized.  The  green  color  used  to  display 
range  and  azimuth  on  the  display  tended  to  blend  with  some 
backgrounds,  making  it  difficult  to  read. 

Map  of  the  Future 

Despite  its  complexity,  the  MOF  (see  Table  4)  was  not 
perceived  as  difficult  to  use  or  to  learn  to  use.  Only  the  Call 
for  Fire  task  received  a  high  difficulty  rating.  Several  factors 
may  account  for  this. 

Table  4. 

Ratings  of  Map  of  the  Future  Tasks  (Means) 


Task 

Difficulty 

Frequency 

Learning 

Difficulty 

Call  for  fire 

3.21 

2.73 

2.71 

Understanding  map 
graphics 

2.68 

4.36 

NA 

Make  a  report,  using 
device 

2.65 

2.74 

NA 

Keeping  track  of  things 
on  map 

2.63 

4.53 

2.45 

Reading  the  map 

2.49 

4.47 

NA 

Receive  a  map,  using 
device 

2.40 

2.56 

NA 

Seeing  the  map 

2.13 

4.64 

NA 

Using  the  map 

2.09 

4:46 

NA 

Maintaining  map 
orientation 

1.77 

4.50 

1.80 

Learn  the  menus 

NA 

NA 

2.34 

Respond  to  a  message 

NA 

NA 

2.16 

N  =  24 


Frequency  Key:  1-Never;  2-Occasionally;  3-Moderately  Often;  4- 
Frequently;  5-Almost  Constantly 

Difficulty  and  Learning  Difficulty  Key;  1-Very  Easy;  2-Easy;  3- 
Neither  Easy  nor  Difficult;  4-Difficult;  5-Very  Difficult 
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First,  the  MOF  was  a  conventional  laptop  that  used  a^ 
araphical  user  interface  with  icons  and  pull-down  menus  similar 
to  Windows  or  Macintosh  operating  systems.  Given  the  familiarity 
of  the  participants  with  laptop  computers,  they  have 

easy  to  learn  the  basic  concepts.  Second,  since  the 
in  at  least  two  and  usually  three  duty  stations  during 
excursion,  the  participants  had  sufficient  practice  with  it  to 
become  proficient  in  its  use.  Finally,  the  MOF  was  easily 
modifiable  and  the  software  was  modified  on  several  occasions 
during  the  excursions,  based  on  comments  from  the  participants. 

Treadport 

With  respect  to  the  treadport  (see  Table  5) ,  the  task  of 
doing  something  else  while  walking  was  the  only  task 
considered  difficult.  However,  it  was  not  performed  frequently, 
so  participants  had  little  practice. 

Table  5. 

Ratings  of  Treadport  Tasks  (Means) 


Task 

Difficulty 

Frequency 

Learning 

Difficulty 

Doing  something  else 
while  walking 

4.00 

1.33 

NA 

Maintaining  orientation 

2.33 

3.33 

NA 

Navigate  in  virtual 
environment 

2.33 

2.50 

1 . 63 

Maintaining  balance 

2.33 

2.33 

2.25 

Walking  on  the  Treadport 

2.00 

2.67 

NA 

Change  orientation 

NA 

NA 

2.88 

N  =  4 

Frequency  Key;  1-Never;  2-Occasionally;  3-Moderately  Often;  4- 
Frequently;  5“Almost  Constantly 

Difficulty  and  Learning  Difficulty  Key:  1-Very  Easy;  2-Easy;  3- 
Neither  Easy  nor  Difficult;  4-Difficult;  5-Very  Difficult 


Grunt 

Adjusting  fires  with  the  Grunt  Personal  Data  Assistant  (see 
Table  6)  was  difficult  to  do  and  the  most  difficult  Grunt  task  to 
learn  to  do.  However,  this  task  was  not  performed  frequently. 
Overall,  the  Grunt  was  considered  easy  to  use.  Some  problems 
using  the  Grunt  did  surface  in  the  group  interviews.  The  primary 
problem  was  a  lack  of  any  indication  that  a  message  had  been 
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Table  6. 


of  Girunt  Tasks  (Means) 


Task 

Difficulty 

Frequency 

Learning 

Difficulty 

arfinst  fires  with  PDA 

3.25 

1.37 

2 . 93 

ComDOsincf  text  messages 

2.63 

2.06 

2.72 

Making  PDA  do  what  you 

wpi  n  t  pd 

2.30 

3.87 

2 .28 

Understanding  PDA  menu 

2.11  ^ 

4.06 

1.89 

Remembering  PDA 
functions  1 

2.08 

3.67 

2 . 17 

Understanding  PDA 
s  vitibols 

2.06 

3.53 

NA 

Seeing  PDA  display 

2.03 

4.17 

NA 

Realizing  a  message  is 
waiting 

1.72 

3.14 

2 . 17 

N  =  14 


Frequency  Key:  1-Never;  2-Occasionally;  3-Moderately  Often;  4 
Frequently;  5-Almost  Constantly 


Difficulty  and  Learning  Difficulty  Key: 
Neither  Easy  nor  Difficult;  4~Difficult, 


1-Very  Easy;  2-Easy; 
5-Very  Difficult 


sent,  received  by  another  participant,  or  that  a  request  for  fire 
had  been  acted  upon.  The  last  was  not  a  Grunt  problem  per  se, 
but  r'Lcted  the  Initial  lack  of  a  detailed  SOP  (the  FDC  ahould 
indicate  when  a  request  for  fire  has  been  filled),  and  the 
workload  placed  on  the  role  players  in  the  simulation. 


Dial-A-Tank  Surrogate  Vehicle 

The  Dial-A-Tank  (see  Table  7)  was  considered  easy  to  use  and 
to  learn  to  use. 


System  Issues 

System  issues  are  related  to  the  general  characteristics  of 
the  system,  simulation,  or  setting,  or  to  multiple  items  of 
equipment.  They  include  non-equipment  specific  tasks,  system 
integration,  suggested  system  improvements,  and  team  organization 
and  workload.  Data  on  systems  issues  come  from  the 
interview,  observer  notes,  and  to  a  lesser  extent,  the  few 
questionnaire  items  that  were  not  related  to  items  of  equipment. 
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Table  7. 


Ratings  of  Dial-A-Tank  Tasks  (Means) 


Tasks 

Difficulty 

Frequency 

Learning 

Difficulty 

Steering  vehicle 

2.13 

2.38 

NA 

Maintaining  orientation 

1.75 

3.00 

1.93 

Navigate  in  virtual 
environment 

1.63 

2.00 

2.07 

Accelerating  vehicle 

1.50 

2.00 

NA 

Braking  vehicle 

1.50 

2.00 

NA 

Stopping  Vehicle 

1.50 

2.00 

NA 

Drive  the  Vehicle 

NA 

NA 

2.07 

N  =  7 


Frequency  Key;  1-Never;  2-Occasionally;  3-Moderately  Often;  4- 
Frequently;  5-Almost  Constantly 

Difficulty  and  Learning  Difficulty  Key:  1-Very  Easy;  2-Easy;  3- 
Neither  Easy  nor  Difficult;  4-Difficult;  5-Very  Difficult 


Non-equipment-Specific  Tasks 

The  most  difficult  of  the  non-equipment-specific  tasks  (see 
Table  8)  was  keeping  track  of  multiple  targets,  a  result  that  was 
also  confirmed  in  the  group  interview.  Closely  related  was  the 
task  of  remembering  to  make  corrections.  Despite  the  low 
difficulty  of  communications,  participants  reported  in  the  group 
interview  that  the  simulation  of  radio  traffic  was  distracting. 
The  voice-actuated  microphones,  intended  to  carry  traffic  among 
the  elements  (station  and  team  leader,  station  and  fire  direction 
center,  etc.)  also  tended  to  pick  up  communications  intended  for 
other,  physically  co-located  members  at  the  station.  Some 
participants  found  this  distracting.  Despite  the  need  of  the 
operator  of  the  simulated  radio  to  listen  to  both  radio  and  local 
communications,  the  earpieces  on  the  headphones  completely 
covered  both  ears.  Thus  participants  tended  to  wear  them  at  an 
angle,  leaving  one  ear  uncovered.  This  eventually  became 
uncomfortable.  Either  a  single-ear  headphone  or  smaller 
earpieces  that  do  not  cover  the  ear  would  have  alleviated  the 
problem. 


Table  8. 


Ratings  of  Non-Equipment-Specific  Tasks  (Means) 


Task 

N 

Frequency 

Difficulty 

Keeping  track  of 
multiple  targets 

33 

3.84 

3.21 

Doing  multiple  tasks 
simultaneously 

26 

3.61 

2.88 

Remembering  to  make 
corrections 

29 

2.29 

2.77 

Switching  among  displays 

23 

3.08 

2.49 

Maintaining  situational 
awareness 

30 

4.06 

2.47 

Communicate  (Team  & 
Higher  Headquarters) 

24 

3.48 

2.19 

Coordinate  with  team 
member 

24 

4.45 

11  ^  W 

2.17 

_ j r\  4 

r  •  -I-  V  wo- ^  - - - j  ,  - 

Frequently;  5-Almost  Constantly 

Difficulty  Key:  1-Very  Easy;  2-Easy;  3-Neither  Easy  nor 
Difficult;  4-Difficult;  5-Very  Difficult 


System  Integration 

Since  the  simulation  was  created  largely  by  building  on 
existing  equipment,  it  is  not  surprising  that  some  instances  of 
incompatibility  and  incomplete  integration  of  components  were 
observed.  There  was  no  way  to  hand  off  a  target  from  the  MOF  to 
the  Grunt.  Early  in  the  week,  the  MOF  showed  direction  to 
targets  in  degrees,  while  the  COVER  and  Melios  reported  viewing 
direction  in  mils.  Changing  the  MOF  readout  to  mils  was  simple 
and  could  be  done  using  the  MOF  user  menus.  However,  until 
someone  who  knew  how  to  make  the  change  was  made  aware  of  the 
need  for  it,  the  participants  used  plastic , protractors,  marked  in 
both  mils  and  degrees,  as  a  job  aid  to  make  the  conversion. 

Another  problem  that  participants  noted  was  the  difficulty 
of  correlating  the  various  terrain  representations .  The  paper 
map,  MOF,  ModSAF,  and  the  visual  displays  (out  the  window,  COVER, 
and  Melios)  each  used  similar  but  slightly  different 
representations . 

Team  Organization  and  Workload 

One  of  the  critical  issues  related  to  team  organization  is 
the  role  of  the  team  leader.  While  he  had  overall  responsibility 
for  the  functioning  of  the  targeting  elements,  he  was  often  out 
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of  th©  information  loop.  He  did  not  prioritize  targets  or  fire 
reguests  and  did  not  know  when  subordinates  reguested  fires  using 
the  Grunt.  Despite  this,  the  participants  felt  that  the  leader 
had  too  much  to  do  to  also  be  a  part  of  a  subordinate  element. 
(These  excursions  did  not  address  other  team  members  responsible 
for  security,  logistics,  etc.)  The  most  efficient  structure  of 
the  targeting  element  of  a  team  was  found  to  be  three  persons  -- 
Leader/MOF  Operator,  a  Vehicle  Driver/COVER  Operator  and  Grunt 
Operator . 

Workload  during  the  excursions  varied  from  long  periods 
during  which  no  targets  were  visible  to  periods  during  which 
multiple  weapons  strikes  were  in  flight  against  enemy  targets. 
Participants  reported  difficulty  managing  the  latter  situation. 

It  was  difficult  to  remember  which  mission  was  being  fired 
against  what  target,  and  where  the  target  had  been  and  was  likely 
to  be  when  the  munitions  arrived  and  designation  (for  a  smart 
weapon)  or  battle  damage  assessment  was  likely  to  be  reguired. 
Even  with  the  viewing  elevation  provided  by  the  COVER,  a  target 
was  unlikely  to  remain  in  continuous  view  for  the  duration  of  a 
fire  mission.  It  tended  to  disappear  behind  a  woodline  or 
terrain  elevation.  If  multiple  targets  were  thought  to  be  in  the 
area,  the  COVER  would  search  for  additional  targets  after  a  fire 
mission  had  been  reguested,  and  then,  at  the  designated  time,  try 
to  relocate  the  first  target. 

As  the  number  of  targets  increased,  so  did  the  complexity  of 
this  operation.  One  group  used  paper  and  pencil  to  record  target 
number,  type,  and  last  known  location.  The  only  automated 
support  in  this  operation  was  an  algorithm  that  predicted  target 
location  based  on  a  straight  line,  constant  speed,  extrapolation 
from  two  consecutive  lasings.  Unfortunately,  land  combat 
vehicles  rarely  travel  in  a  straight  line  at  a  constant  speed  for 
the  duration  of  a  10  to  20  minute  engagement  cycle. 

Suggested  System  Improvements 

The  system  improvements  that  the  participants  suggested 
focused  primarily  on  ways  to  help  them  deal  with  periods  of  high 
workload.  The  first  was  the  development  and  use  of  "fire  and 
forget"  weapons  or  ones  with  automatic  tracking  sensors .  A  less 
sophisticated,  partial  solution  would  be  to  provide  a  capability 
on  COVER  to  memorize  and  return  to  a  previous  location,  much  like 
a  "last  channel"  button  on  a  TV  remote.  Another  solution  would 
be  to  provide  a  way  to  indicate,  on  the  fire  reguest,  that  you  do 
not  expect  to  be  able  to  maintain  line  of  sight  to  a  target. 

This  would  alert  the  FDC  not  to  fire  a  weapon  that  reguired 
designation.  A  final  proposed  system  improvement  was  the 
addition  of  thermal  and  infrared  capability  to  the  COVER. 
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Situational  Awareness 


It  appears  that  the  use  of  the  various  technological  items 
both  increased  and  decreased  participant  situational  awareness. 

By  providing  an  integrated  picture  of  the  friendly  and  the  enemy 
situation  (as  could  be  determined  from  various  intelligence 
assets),  they  provided  good  "global"  situational  awareness,  i.e., 
the  big  picture.  However,  by  focusing  their  attention  on  a  few 
display  screens  during  periods  of  high  workload  and  stress  they 
decreased  "local"  situational  awareness.  Several  instances  were 
observed  in  which  enemy  were  in  clear  view  on  the  "out  the 
window"  displays,  but  were  not  detected.  In  these  high  workload 
situations,  participants  tended  to  focus  attention  on  their 
device  displays,  and  paid  little  attention  to  the  simulated  out 
the  window"  view.  This  is  a  common  phenomenon  called 
"attentional  narrowing"  (Endsley,  1995) . 

There  were  numerous  reasons  for  this  that  might  not  apply  in 
combat.  First,  given  the  tactical  situation,  the  out-the-window 
view  was  a  low  priority  information  source.  The  soldiers  needed 
to  attack  the  targets  beyond  unaided  visual  range.  Second, 
close-in  targets  were  rare,  so  there  was  little  reinforcement  for 
using  the  out-the  window  view.  Third,  the  COVER  could  be  used  to 
provide  the  same  information  with  less  disruption  of  ongoing 
activities.  Fourth,  the  low  resolution  of  the  simulation  made  it 
difficult  to  detect  targets  with  unaided  vision.  Finally,  by  the 
time  a  target  was  detected  with  unaided  vision,  it  was  too  late 
to  call  in  fire.  No  munitions  could  arrive  in  time  and  the  team 
had  no  organic  firepower. 

Fatigue  and  Side  Effects 

The  results  of  the  post-immersion  questionnaire  are  shown  in 
Appendix  I.  There  was  a  slight  tendency  for  the  participants  to 
report  more  immersion  as  they  became  more  familiar  with  the 
system.  Scores  for  the  second  administration  (day  4)  were 
generally  higher  than  those  for  the  first  Xday  2) .  There  was 
also  a  wide  range  of  responses  to  most  of  the  items,  reflecting 
substantial  individual  differences  in  how  the  participants 
responded  to  the  simulation.  The  general  tone  of  the  responses 
was  that  they  were  able  to  become  involved  in  the  virtual 
environment  experience,  adjusted  quickly  to  it,  became  at  least 
reasonably  proficient  in  moving  and  interacting  with  it,  and 
most,  at  least  occasionally,  felt  completely  focused  on  the  task 
or  environment. 

The  pre-  and  post-simulation  self-ratings  on  simulator 
sickness  are  presented  in  Table  9.  (Symptoms  for  which  there 
were  only  ratings  of  "none"  are  omitted  from  the  table,  as  is  the 
unused  "severe"  rating.)  This  table  provides  a  comparison  of  the 


21 


baseline  (pre)  and  the  post-ratings.  Initially  only  the  symptom 
blurred  vision  was  rated  as  high  as  "moderate;"  in  all  other 
instances  symptoms  were  rated  "none"  or  "slight."  In  the  post¬ 
simulation  ratings,  of  the  16  symptoms,  8  were  rated  at  least 
once  as  slight  or  moderate.  Slight  fatigue  was  the  most 
frequently  reported  symptom,  followed  by  eyestrain,  fullness  of 
the  head,  and  blurred  vision.  The  one  instance  of  moderately 
blurred  vision  was  reported  by  the  same  individual  who  had 
reported  that  problem  on  the  pre-simulation  questionnaire  for 
that  day. 

Participants  reported  in  the  group  interview  that  they  felt 
they  would  be  unable  to  perform  this  type  of  activity  for  8-10 
hours.  Participation  in  the  simulation  produced  some  symptoms  of 
discomfort  among  the  participants.  Fatigue,  eyestrain,  and  a 
feeling  of  pressure  in  their  head  were  common  symptoms.  No  one 
considered  symptoms  to  be  severe. 


Table  9 

Pre-  and  Post-Simulation  Self-Ratings  of  Severity  of  Simulator 
Sickness  Symptoms  (Percent) 


Symptom 

%  None 

%  Slight 

%  Moderate 

Pre 

Post 

Pre 

Post 

Pre  Post 

Fatigue 

79 

37 

21 

62 

Eye  strain 

96 

54 

4 

37 

0  8 

Fullness  of  the  head 

92 

67 

8 

33 

Blurred  vision 

92 

88 

4 

8 

4  4 

General  discomfort 

96 

87 

4 

12 

Headache 

96 

100 

4 

0 

Dizzy  (eyes  open) 

100 

96 

0 

4 

Difficulty  concentrating 

92 

96 

8 

4 

Difficulty  focusing 

92 

96 

8 

4 

Personnel,  Skill,  and  Training  Requirements 

Participants  were  asked  what  prior  experience  or  training 
helped  in  these  exercises.  One  factor  noted  was  computer 
experience.  As  discussed  earlier,  all  participants  reported  at 
least  some  experience  with  laptop  computers,  and  four  of  the  six 
reported  experience  sending  and  receiving  E-mail.  Although  they 
were  not  asked  about  experience  with  computers  in  general,  it  was 
obvious  that  they  were  quite  familiar  with  the  basics  of 
interacting  with  computers:  cursor  control,  icons,  pull-down 
menus,  and  display  windows.  A  related  factor  was  experience  with 


coinputer  games.  Participants  saw  this  as  highly  relevant.  Five 
reported  some  computer  game  playing,  and  three  indicated  more 
than  occasional  game  playing.  For  actual  deployment  of  an  STP21 
team,  more  computer  expertise  would  be  required.  Soldiers  would 
not  only  have  to  know  how  to  use  the  systems,  but  also  to 
diagnose  and  correct  hardware  and  software  problems  that  might 
arise . 

Participants  also  reported  that  prior  leadership  experience 
and  tactical  experience  were  beneficial.  Forward  observer 
training  was  considered  largely  irrelevant  because  current  FO 
procedures  are  not  applicable  to  the  future  digital  environment' s 
long  range  engagement  cycles  for  remotely  delivered  weapons, 
although  it  did  provide  familiarity  with  basic  concepts. 

Participants  reported  that  they  began  to  feel  comfortable 
with  the  equipment  and  their  roles  by  the  end  of  the  second  or 
the  beginning  of  the  third  day.  However,  it  was  clear  from  their 
comments  and  questions  during  AARs  and  from  observer  notes  that 
they  were  still  learning  after  this  time.  Some  training  was 
actually  conducted  on  the  morning  of  the  fourth  day.  They 
expressed  a  need  for  additional  and  more  structured  training, 
with  specially  designed  training  scenarios  and  a  "coach"  to 
provide  advice.  They  noted  that  most  computer  games  provide 
training  scenarios. 


Discussion 

Overview 


Within  certain  limits  the  concept,  and  the  simulation  of 
that  concept,  worked  quite  well  as  the  flexibility  inherent  in 
the  simulation  permitted  modification  both  to  scenarios  and 
equipment.  The  EEAs  were  addressed,  if  informally. 

The  first  objective  or  EEA  dealt  with  small  team  situational 
awareness,  whether  devices  assisted  in  utilization  of  sensor 
information,  and  the  added  value  for  each  kind  of  sensor.  The 
second,  related,  EEA  explored  enhanced  combat  effectiveness, 
through  bringing  accurate  and  timely  fires  in  support  of 
neighboring  teams  and  a  higher  headquarters. 

The  overall  concern  was  whether  the  teams  could  utilize 
information  received  from  sensors  and  the  various  PDAs .  Although 
effectiveness  varied,  the  teams  were  able  to  perform  the  required 
tasks.  Both  information  receipt  and  use  were  demonstrated. 

Measures  of  performance  included  the  ability  to  service 
targets,  and  the  timeline  from  presentation  through  calls  for 
fire  and  adjustment  if  needed.  Timelines  were  difficult  to 
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obtain,  due  primarily  to  difficulties  with  the  prototype 
equipment.  Also  of  interest  was  team  survivability  —  whether 
they  could  engage  before  they  were  themselves  engaged.  Teams 
were  occasionally  overrun,  perhaps  due  to  the  tendency  to  become 
focused  on  the  device  they  were  using  rather  than  the  situation. 

Another  concern  was  the  amount  of  terrain  over  which  any  one 
team  could  be  expected  to  maintain  observation  —  the  maximum 
limits  of  awareness.  Generally,  a  360  degree  area  of 
responsibility  proved  too  great.  A  balanced  team  has  roles  and 
responsibilities  allocated  to  maximize  combat  effectiveness. 

While  individual  situational  awareness  should  be  enhanced,  combat 
power  is  enhanced  through  team  work. 

The  third  EEA  focused  on  the  simulation  as  a  concept 
exploration  tool.  This  included  the  transparency  of  the 
simulation  to  the  players  —  their  ability  to  accept  it  and  the 
prototype  equipment,  and  finally,  the  capability  of  the 
simulation  to  provide  meaningful  data.  The  simulation  provided 
useful  information.  Some  training  issues  (prototype  equipment, 
practice  effects)  and  simulator  issues  (sickness,  artificiality) 
were  potential  distracters.  Difficulty  with  equipment  was 
undeniable  (the  COVER  joystick),  and  practice  effects  were 
apparent  (MOF) ;  they  could  be  overcome  with  better  equipment  and 
more  training  time.  For  the  duration  of  these  exercises, 
simulator  sickness  was  not  an  issue;  the  artificiality  of  the 
environment  was  of  little  impact,  and  may  have  lessened  any  ill 
effects . 

The  simulation  fidelity  was  adequate  for  representing  the 
critical  role  that  the  small  team  can  play  in  dealing  with 
incomplete,  imprecise,  or  erroneous  information  from  electronic 
and  other  remote  sensors.  However,  limitations  in  visual 
fidelity  made  it  difficult  to  distinguish  military  from  civilian 
entities.  Therefore,  the  simulation  could  not  demonstrate  a 
benefit  of  having  observers  on  the  ground:  avoiding  collateral 
damage  to  civilians  and  civilian  structures. 

Lessons  Learned:  Equipment 

It  is  fully  acknowledged  that  the  specific  equipment  used  in 
the  simulation  is  different  from  that  which  would  be  found  on  the 
battlefield  of  the  year  2010.  However,  certain  lessons  learned 
merit  discussion. 


Treadport .  The  scenarios,  as  run,  provided  no 
to  take  advantage  of  the  unique  capabilities  of  the 
No  observation  points  were  accessible  only  on  foot, 
observation  from  a  prone  position  was  not  possible. 


opportunities 
treadport . 
and 
When 
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movement  was  required  to  obtain  or  maintain  line  of  sight  to  a 
target,  it  had  to  be  accomplished  more  rapidly  than  could  be  done 
by  foot.  Questions  remain  about  the  value  added  of  this  concept 
in  this  domain. 

Dial-A-Tank.  This  surrogate  was  rarely  used  for  mobility. 
When  it  was  used,  it  was  apparent  that  the  soldiers  were  having 
difficulty  in  maintaining  orientation.  Additionally,  there  was  a 
tendency  to  drive  using  the  MOF  display  rather  than  the  terrain 
display. 

Map  of  the  Future.  The  original  concept  was  for  the  MOF  to 
be  used  only  as  a  map;  in  practice  it  was  used  to  call  for  and 
receive  fire  missions,  and  for  E-mail.  The  MOF  was  clearly  not 
designed  for  these  missions.  The  map  scale  was  not  suitable  for 
the  size  of  the  area  available;  preformatted  messages  were  not 
well  thought  out.  Provisions  were  not  made  for  feedback  on 
requests  for  fire  and  targeting  effects,  but  need  for  feedback 
was  clearly  demonstrated. 

Melios.  The  current  configuration  of  Melios  may  have 
precluded  seeing  the  advantages  of  this  device.  Besides  the 
previously  noted  weight  problems,  and  the  difficulty  of  using 
Melios  while  on  a  treadmill,  Melios  would  be  most  useful  in  flat 
and  desert-like  terrain,  rarely  available  in  this  scenario. 

COVER .  While  the  concept  appears  viable,  the  specific 
implementation  (joystick  control)  does  not.  The  joystick  was 
often  unresponsive,  erratic,  and  could  not  be  held  steady.  It 
impacted  on  the  operators'  abilities  to  use  the  device 
effectively.  A  fielded,  future  version  would  respond  better,  and 
could  have  locations  (grids,  direction,  azimuth)  displayed 
directly  on  the  MOF. 

Grunt .  The  consensus  was  that  this  device  was  in  many 
respects  redundant  to,  and  less  useful  than,  the  MOF.  Having  two 
systems,  with  different  conventions  and  key  strokes,  is 
inefficient.  A  single  device  would  reduce  training  time  and 
COVER  and  Melios  could  be  linked  directly  to  the  MOF  rather  than 
to  the  Grunt. 

Lessons  Learned;  Personnel,  Organization,  Workload 

Three  person  groups  seemed  most  useful,  with  the  team  leader 
assisting  by  filtering  information  (nice  to  know  vs.  need  to 
know) .  The  groups  that  appeared  to  do  well  assisted  each  other 
and  developed  SOPs.  Within-group  communication  and  cooperation 
are  critical  —  groups  need  to  work  together  to  service  targets, 
and  frequently  can  share  in  the  identification  process.  They 
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also  need  a  predetermined  between-group  plan  for  target  hand  off, 
if  a  target  passes  from  one  sector  to  another. 

The  tendency  of  the  participants  to  lose  local  situational 
awareness,  particularly  in  high  stress  situations,  points  out  the 
need  for  a  separate  element  of  the  team  to  maintain  loca 
security.  The  fatiguing  nature  of  the  tasks  suggests  a  need  for 
multiple  shifts  of  operators » 


Workload  and  performance  issues  are  largely  personality 
driven.  There  were  apparent  differences  between  the  groups  where 
there  was  almost  no  chatter,  and  those  where  it  was  common.  When 
there  was  little  to  do,  each  of  the  soldiers  acted  bored  or 
complacent  and  occasionally  lost  focus.  Although  some  did  better 
with  pressure,  others  were  sometimes  apparently  overloaded.  The 
small  remotely  placed  team  may  require  a  special  kind  of  personal 
discipline  to  ensure  that  attention  and  awareness  are  maintaine 
throughout  a  period  of  inactivity.  Some  personnel  selection  and 
qualification  issues  will  have  to  be  addressed. 

Lessons  Learned:  Observer  and  Participant  Training 


Training  is  critical  for  equipment,  tactics,  techniques  and 
procedures.  It  must  include  not  only  learning  to  use  new 
equipment,  but  a  chance  to  practice  or  experiment  with  it. 
Observers  must  also  be  trained,  both  on  the  hardware  and  on 
observation  techniques.  Observers  need  to  h^ar  team  cross  talk 
and  radio  traffic,  and  share  the  soldier's  field  of  view.  Only 
in  this  way  can  an  observer  evaluate  operator  performance.  For 
example,  the  Grunt  observer  monitor  was  not  useful  because  the 
process  of  creating  messages  prior  to  their  display  on  the  screen 
was  not  visible.  The  same  observers  also  need  to  be  available 
every  day  to  note  posture  changes,  signs  of  fatigue,  eyestrain, 
etc.,  and  to  get  a  feel  for  the  simulation  and  for  player 
personalities . 


Although  radio  traffic  should  be  downplayed,  personnel  need 
instruction  on  communication  procedures,  call  signs,  etc.  For 
example,  a  training  deficiency  became  apparent  with  the  radio's 
voice  activated  mike.  Only  the  experienced  soldiers  knew  to 
start  messages  with  double  words  ("shooter,  shooter  ) ,  the  first 
word  to  activate  the  mike,  the  second  to  start  the  message. 


Just  as  there  are  conventions  for  use  of  radios,  soldiers 
need  SOPs  (acronyms,  abbreviations,  formats)  to  be  used  in  the 
simulation.  Perhaps  the  most  important  skill  requirement  for 
team  members  will  be  a  relatively  high  degree  of  computer 
literacy.  All  of  the  devices  were  slower  to  perform  than  typical 
office  equipment.  Operator  unfamiliarity  with  system  speed 
caused  extra  work,  either  through  resending  messages,  or 
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restriking  keys.  For  example,  maps  required  a  finite  time  to 
recreate  when  they  were  moved  —  impatient  keystrokes  did  not 
speed  up  the  system.  Variability  in  operator  performance  and 
efficiency  might  also  be  attributed  to,  for  example,  differing 
typing  ability  levels.  Additional  training  would  alleviate  some 
of  these  problems  and  improve  system  performance. 

Scanning  and  searching  for  soft-skinned  or  personnel  targets 
at  short  ranges  in  a  limited  sector,  and  preparing  for  direct 
fire  weapon  engagements,  do  not  prepare  the  soldier  for  the  wide 
and  long  range  area  search  required  in  this  simulation.  They 
lacked  scanning  and  searching  techniques,  and  did  not  know  about 
being  systematic,  using  patterns,  or  varying  power  to  get  better 
or  different  perspectives.  Training  such  as  is  available  for 
Scouts  or  long  range  surveillance  units  (LRSU)  may  be  required. 

Small  team  members  need  cross  training  on  all  of  the 
devices.  This  is  mandatory  as  a  casualty  backup  plan,  but  also 
helps  with  work  and  rest  plans,  boredom,  and  stress  reduction.  A 
single  device  (like  the  MOF)  which  can  accomplish  multiple 
functions  and  can  be  distributed  to  each  of  the  players,  may  be  a 
way  of  sharing  workload  while  maintaining  efficiency.  SOPs  are 
critical;  job  aids  may  help  in  keeping  track  of  information. 

Lessons  Learned:  Simulations 


Some  of  the  most  valuable  lessons  from  simulation  this  far 
in  advance  of  development  do  not  have  to  do  with  the  man-machine 
interface  of  specific  items  of  equipment  (keyboard  layouts  or 
menus) ,  but  with  interfaces  among  the  various  systems  and  the 
load  they  put  on  the  humans  in  the  loop  (manually  transcribing 
information  from  one  electronic  system  to  another) .  Fifteen 
years  ago,  only  a  few  computer  scientists  would  have  been 
familiar  with  the  windows-based  interface  used  in  these  systems. 
Today,  it  is  widely  understood.  Whatever  the  advances  in 
computer  interface  technology  over  the  next  fifteen  years,  it 
seems  certain  that  they  will  render  obsolete  any  interface 
changes  made  today.  However,  basic  human  capabilities  and 
limitations  will  remain  unchanged,  and  systems  designs  will  have 
to  allocate  functions  between  humans  and  computers  in  a  way  that 
best  takes  advantage  of  those  capabilities  and  limitations. 

The  exercises  demonstrated  that  simulations  can  contribute 
to  the  development  of  new  concepts  in  the  following  ways : 

Improving  equipment  design.  Using  simulation  for 
development  helps  improve  the  design  of  equipment  and  systems. 

The  simulation  identified  operator  interface  capabilities  and 
problems  with  each  piece  of  equipment,  and  the  impact  on  operator 
workload.  As  in  the  STP21  demonstration,  this  can  occur  even  if 
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it  is  not  possible  to  obtain  detailed  quantitative  data  on 
operator  performance.  The  simulation  afforded  the  opportunity  to 
make  changes  to  items  of  hardware  and  software  during  the 
exercises . 

Demonstrating  concepts.  Simulation  can  be  used  to 
demonstrate  concepts,  as  long  as  participants  know  the  intent  and 
limitations  of  the  simulation.  Participant  training  is  required 
to  ensure  that  the  simulation  will  be  able  to  show  the  value  of 
each  piece  of  equipment. 

Placing  humans  in  the  loop.  It  also  showed  the  value  of 
putting  the  human  back  in  the  loop.  Modeling  can  quickly  be 
invalidated  by  individual  differences  in  performance.  Operators 
take  shortcuts  and  make  mistakes. 

Increasing  innovation.  Soldiers  in  the  field  environment 
are  often  hampered  by  what  they  already  know  ("we  always  do  it 
this  way")  because  there  is  a  frame  of  reference  for  everything. 
In  a  simulation,  reality  is  suspended,  and  participants  may  be 
more  open  to  new  ideas,  often  on  some  very  basic  things. 

Simulator  sickness  was  not  a  serious  problem.  The  SSQ 
scores  were  lower  than  levels  observed  after  initial  training 
with  a  tank  driver  simulator  (Lampton  et  al . ,  1995)  and  in 
experiments  in  virtual  environments  using  helmet  mounted  displays 
(Lampton  et  al.,  1994).  The  symptoms  observed  here  may  have 
resulted  from  several  factors.  Any  simulator  that  portrays  self- 
motion  may  produce  simulator  sickness.  In  both  the  WISE  and  the 
vehicle  screens,  motion  was  depicted  periodically  and  was 
observable  by  all  the  team  members  at  each  particular  station, 
not  just  the  person  walking  or  running  on  the  WISE,  or  driving 
the  vehicle.  Also,  because  of  the  low  level  of  illumination  of 
the  displays  and  ambient  light,  displays  were  frequently 
difficult  to  view.  This  may  have  contributed  to  eye  strain  or 
fatigue . 

Recommendations  for  Future  Excursions 


Part  of  the  value  of  this  simulation  is  in  learning  how  to 
do  it  better  another  time.  If  these  excursions  were  repeated, 
with  different  equipment  characteristics,  the  results  might  be 
very  different. 

From  a  system  design  standpoint,  the  equipment  that  supports 
STP21  must  be  an  integrated  system.  The  human  should  not  be  in 
the  role  of  the  conduit  passing  information  from  one  component  to 
another.  Effective  information  management  requires  feedback. 
Units  need  reporting  SOPs .  Update  reports  at  random  intervals 
overload  even  experienced  personnel. 


28 


Despite  attempts  to  use  digital  communication  throughout, 
voice  communication  was  faster,  more  efficient,  and  probably 
caused  fewer  errors.  The  soldiers  relied  on  the  familiar  system, 
the  one  from  which  they  received  immediate  and  reliable  feedback. 
Feedback  needs  to  be  designed  into  the  digital  communications  as 
well . 


Conventions  are  needed  for  tracking  messages.  To  assist  in 
information  management.  E-mail  must  indicate  senders,  priorities, 
and  the  time.  Changes  needed  are  better  ways  to  accept,  send, 
save  and  delete  messages,  as  well  as  a  better  way  to  display 
them.  Some  preformatted  messages  are  mandatory. 

There  must  be  a  way  to  designate  or  identify  a  potential 
target  and  follow  it,  assessing  speed,  direction,  etc.,  without 
having  to  go  through  a  laborious  device-based  process.  Each  time 
attention  is  diverted  from  the  field  of  view,  potential 
information  is  lost. 

Equipment  changes  need  to  be  directed  toward  saving  time  and 
making  processes  more  efficient,  to  reduce  errors,  leave  fewer 
things  to  memory,  improve  the  operator's  ability  to  use  the 
devices,  and  maintain  situational  awareness.  One  device  should 
be  able  to  combine  the  functions  of  the  Grunt,  the  MOF,  and  any 
kind  of  PDA,  with  only  one  set  of  protocols  to  learn.  This  would 
also  ease  training. 


Conclusions 

Overall,  this  research  project  reinforced  the  view  that 
virtual  simulations  can  greatly  enhance  the  concept  development 
process  by  providing  the  capability  to  assess  mission  performance 
before  a  system  is  fielded.  Modeling  cannot  incorporate  the 
dynamics  of  a  small  team  the  way  a  simulation  does. 

The  STP21  simulation  was  hampered  by  present  day  equipment, 
and  perhaps  by  preconceived  ideas.  Borrowed  equipment  did  not 
link  well  in  the  simulation  environment,  but  simulations  are 
ideal  for  first  steps.  Prototype  equipment,  despite  issues 
related  to  hardiness,  interconnectivity,  and  linking,  is 
characterized  by  the  inherent  ability  to  change  it. 

Other  limitations  of  these  excursions  as  STP21  predictors 
should  be  noted.  First,  equipment  components  were  not  designed 
to  function  as  an  integrated  system,  and  their  future  design 
might  be  very  different.  Procedures  for  employment  were  not 
fully  developed  and  tested,  and  personnel  were  not  thoroughly 
trained.  They  had  not  adequately  learned  and  practiced  the  basic 
procedures,  e.g.,  scanning  and  target  identification.  As  a 
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result,  the  data  collected  probably  underrepresented  performance 
that  might  actually  be  achieved. 

In  the  independent  small  team  environment,  soldier  selection 
and  training  are  critical.  Issues  such  as  workload,  fatigue, 
boredom,  and  stress  need  to  be  addressed.  With  respect  to  task 
overload,  there  clearly  were  times  when  the  soldiers  could  not 
keep  up  with  the  action.  This  was  partially  due  to  the 
awkwardness  of  the  eguipment  but  even  if  it  had  worked  well,  they 
would  have  reached  overload  fairly  quickly  without  an  automated 
fire-and-forget  system. 

The  amount  of  information  available  to  a  team  needs  to  be 
evaluated  and  it  will  be  important  to  make  some  decisions  about 
use  of  technology.  Simply  automating  and  speeding  up  current 
procedures  may  just  make  errors  faster  and  more  frequent.  Just 
because  information  is  available  does  not  guarantee  that  it  is 
useful . 

The  intended  precise  element  by  element  timeline  —  how  long 
it  takes  for  an  observer  to  receive  information,  call  for  fire, 
and  get  munitions  on  a  target  —  was  not  obtained.  But  the 
simulation  showed  that  the  mission  could  be  accomplished  by  the 
soldiers,  and  rough  estimates  derived.  The  concept  of  a  small 
team  portal  appears  viable,  and  the  simulation  facility  proved  to 
be  an  appropriate  place  to  demonstrate  it. 
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LIST  OF  ACRONYMS 


AAR 

ARI 

BDA 

DARPA 

DSB 

EGA 

EEA 

FDC 

FOV 

GPS 

IDA 

ITQ 

LRSU 

MELIOS 

ModSAF 

MOF 

OCS 

OPFOR 

PDA 

PQ 

REMBASS 

RFPI 

SIMNET 

SOP 

SSQ 

STOW-E 

STP 

SUO 

TRADOC 

WISE 


After  Action  Review 
Army  Research  Institute 
Battle  Damage  Assessment 

Defense  Advanced  Research  Projects  Agency 

Defense  Sciences  Board 

Exercise  Control  Area 

Essential  Elements  of  Analysis 

Fire  Direction  Center 

Field  of  View 

Global  Positioning  System 

Institute  for  Defense  Analyses 

Immersive  Tendencies  Questionnaire 

Long  Range  Surveillance  Unit 

Mini  Eyesafe  Laser  Infrared  Observation  Set 

Modular  Semi -Automated  Forces 

Map  of  the  Future 

Officer  Candidate  School 

Opposing  Forces 

Personal  Data  Assistant 

Presence  Questionnaire 

Remotely  Monitored  Battlefield  Sensor  Systems 

Rapid  Force  Projection  Initiative 

Simulation  Network 

Standing  Operating  Procedures 

Simulator  Sickness  Questionnaire 

Synthetic  Theater  of  War  -  Europe 

Small  Team  Portal 

Small  Unit  Operations 

Training  and  Doctrine  Command 

Walk-In  Synthetic  Environment 
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Appendix  A 
Observer  Worksheet 

-  OBSERVER  DATA  COLLECTION  SHEET  -  STP21  Excursions 

Observer  Name _  Date/Time _ 

Location;  STP1  (treadport)  STP2  (diaki-tank)  Leader  Station 
"jogger*  ‘biker*  “walker* 

Excursion  Number _  Trial  Number _  Probe  Number _ 

Data  Base:  mbced _ desert _ urban _  day/night 

Equipment:  Diaki-tank  /  Foot  Pedal  Treadport 

MOF  Grunt  II  PDA 

Cover  w  joystick  GATHwmelios 

STARTEX _  EXDEX  _ 


TIME 

EVENT 

NOTES/OBSERVATIONS 

1 

i 

• 

— 

— 

A-  1 


Appendix  B 

STP21  Questionnaire 
SECTION  I  (background  information) 

1.  Specialty  Code/Branch/MOS  _ 

2 .  Rank  _ 

3.  Time  in  service  Years  _  Months  _ 

4.  What  is  your  current  duty  position? 

5.  Which  military  schools  have  you  attended?  (e.g<,  basic  and  specialty 

courses,  etc.  List  all.) 

6.  Civilian  education:  degree  _  major  _ 

7.  Do  you  have  difficulty  distinguishing  colors  (color  blind,  color 

weak,  etc.)?  Yes  _  No  _ 

8.  Which  of  the  following  duty  positions  have  you  held?  Check  all  that 

apply,  and  as  appropriate,  whether  at  battalion  or  brigade  level. 

TOW/Tank/BFV  Gunner  _  Tank/BFV/LAV  Commander  _ 

Platoon  Leader  _  Specialty  Platoon  Leader  (specify)  _ 

Company  XO  _  Company  Commander  _ 

Staff  Officer  (specify)  _ 

Engineer  officer  _  NBC/Chemical  officer  _  ADAO  _  ■ 

CESO/SIGO  _  LNO/ANGLICO  _  Other  (specify)  _ 

FO  _  FSO  _  FSCOORD  _ 

If  you  have  had  any  experience  or  training  as  a  forward  observer,  or  in 
requesting/calling  for  fires,  please  describe  this  experience  —  duty 
position,  location,  etc. 


9.  Have  you  participated  in  rotations  at  any  CTC?  How  many  -  where? 


10.  Did  you  participate  in  Desert  Shield/Storm?  Yes  _  No 

Describe. 


3.  When  watching  sports,  do  you  ever  become  so  involved  in  the  game 
that  you  react  as  if  you  were  one  of  the  players? 

1  I _ I _ I _ I _ I _ I - ' 

never  occasionally  often 


4.  When  playing  sports,  do  you  become  so  involved  in  the  game  that  you 
lose  track  of  time? 


never"  occasionally  often 


5.  How  well  do  you  concentrate  on  enjoyable  activities? 


NOT  AT  ALL 


1 _ I 

MODERATELY 

WELL 


VERY  WELL 


6.  How  often  do  you  play  arcade  or  video  games?  (OFTEN  should  be  taken 
to  mean  every  day  or  every  two  days^  on  average.) 


NEVER  OCCASIONALLY  OFTEN 

7.  Do  you  ever  become  so  involved  in  doing  something  that  you  lose  all 
track  of  time? 


never  occasiontu^ly  often 

8 .  How  easily  can  you  switch  attention  from  the  activity  in  which  you 
are  currently  involved  to  a  new  and  completely  different  activity? 

1  I  I  I _ I _ I _ I _ I 

not  so  fairly  quite 

easily  easily  easily 

9.  How  often  do  you  play  games  on  computers? 

1  1 _ I _ I _ I _ I _ 1 _ I 

NOT  AT  ALL  OCCASIONALLY  FREQUENTLY 

10.  How  many  different  video,  computer,  or  arcade  games  have  you  become 
reasonably  good  at  playing? 

I  I _ I _ I. _ I _ I _ 1 _ I 

NONE  ONE  TWO  THREE  FOUR  FIVE  SIX  OR  MORE 

11.  Have  you  ever  felt  completely  caught  up  in  an  experience,  aware  of 
everything  going  on  and  completely  open  to  all  of  it? 

1 _ I _ I _ I _ I _ I _ I _ I 

NEVER  OCCASIONALLY  FREQUENTLY 

12.  Are  you  easily  disturbed  when  involved  in  an  activity  or  working  on 
a  task? 


NEVER 


I _ 1 _ I  I _ 

OCCASIONALLY  OFTEN 
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Appendix  C 

Health  Questionnaire 


1. 

Are  you  in  your  usual  state  of  alertness? 

YES 

NO 

2. 

Do  you  have  any  illness? 

YES 

NO 

If  yes,  please  state  the  nature  of  your  illness  (flu,  cold, 
headache,  sinus  problems,  etc.)* 


3.  Please  indicate  medications  you  have  used  in  the  past  24  hours: 

(You  need  not  include  prescription  medication  routinely  taken  on  a 
daily  basis . ) 


(a) 

NONE 

(b) 

Sedatives  or  tranquilizers 

(c) 

Aspirin,  Tylenol,  other  analgesics 

(d) 

Anti-histamines 

(e) 

Decongestants 

(f) 

Other  (specify) : 

4. 

How 

many  hours  of  sleep  did  you  get  last 

night? 

_  (Hours) 

5, 

Was 

this  amount  sufficient? 

YES 

NO 

6. 

Did 

you  do  any  PT  in  the  last  24  hours? 

YES 

NO 
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Syitiptom  Checklist 

Instructions:  Please  indicate  the  severity  of  symptoms  that  apply 

to  you  right  now  by  circling  the  appropriate  word. 


1.  General  discomfort 

None 

Slight 

Moderate 

Severe 

2 .  Fatigue 

None 

Slight 

Moderate 

Severe 

3 .  Headache 

None 

Slight 

Moderate 

Severe 

4.  Eye  Strain 

None 

Slight 

Moderate 

Severe 

5.  Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

6.  Salivation  increased 

None 

Slight 

Moderate 

Severe 

7 .  Sweating 

None 

Slight 

Moderate 

Severe 

8 .  Nausea 

None 

Slight 

Moderate 

Severe 

9.  Difficulty  concentrating 

None 

.  Slight 

Moderate 

Severe 

10.  *" Fullness  of  the  Head" 

None 

Slight 

Moderate 

Severe 

11.  Blurred  Vision 

None 

Slight 

Moderate 

Severe 

12.  a.  Dizziness  with  eyes 

open 

None 

Slight 

Moderate 

Severe 

b.  Dizziness  with  eyes 

closed 

None 

Slight 

Moderate 

Severe 

13.  Vertigo 

None 

Slight 

Moderate 

Severe 

14.  **Stomach  awareness 

None 

Slight 

Moderate 

Severe 

15.  Burping 

-  None 

Slight 

Moderate 

Severe 

*  Fullness  of  head  is  a  sensation  of  pressure  in  the  head,  just  short  of 
a  headache. 

**  Stomach  awareness  is  usually  used  to  indicate  a  feeling  of  discomfort 
which  is  just  short  of  nausea. 


D-1 


-  ■  .'".'V" 


Appendix  E 

Human  Factors  Equipment  and  Functions  Questionnaire 

For  each  of  the  following  areas,  please  use  the  following  three  scales. 
Indicate  your  answer  in  the  table  below. 

How  frequently  did  you  do  this  during  the  last  scenario?  (Frequency) 

1  Never 

2  Occasionally 

3  Moderately  Often 

4  Frequently 

5  Almost  constantly 

How  difficult  was  it  to  use  the  equipment  or  perform  the  task?  (Do) 

1  Very  Easy 

2  Easy 

3  Neither  Easy  nor  Difficult 

4  Difficult 

5  Very  Difficult 


FREQUENCY 

DO 

Maintaining  situational  awareness 

Switching  among  displays  -  from  MOF  to 
PDA  to  big  screen,  etc. 

Keeping  track  of  multiple  targets 

Remembering  to  make  last  minute 
corrections  per  FSO  instructions 

Doing  multiple  tasks  simultaneously 

Dial  a  Tank  Surrogate  Vehicle 

Accelerating  vehicle 

Braking  vehicle 

Stopping  vehicle 

Steering  vehicle 

Maintaining  orientation 

Navigate  in  virtual  environment 

Treadport 

Walking  on  the  treadport 

Maintaining  orientation 

Navigate  in  virtual  environment 

Maintaining  balance 

Doing  something  else  while  walking 

Map  of  the  Future 

Maintaining  map  orientation 

Seeing  the  map 

Understanding  map  graphics 

Keeping  track  of  things  on  map 

Reading  the  map 

Using  the  map 
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Receive  a  report,  using  device 


Call  for  fire 


Make  a  report,  using  device 


Grunt  II  PDA  _ 


Seeing  PDA  display _ _ _ 


Understanding  PDA  menu _ _ 


Realizing  a  message  is  waiting 


Remembering  PDA  functions _ _ 


Understanding  PDA  symbols _ 


Composing  text  messages _ _ 


Making  PDA  do  what  you  wanted 


Adjust  fires  with  PDA 


GATH/Melios 


Detect  targets  _ _ 


Identify  targets  (name  them) 


Prioritize  targets 


Designate  targets  _ 


,  Distinguish  friendly  &  neutral 


Perform  BDA  _ 


Lazing 


COVER 


Detect  targets  _ _ 


Identify  targets  (name  them) 


Prioritize  targets  _ 


Designate  targets _ 


Distinguish  friendly  &  neutral 


Perform  BDA 


Using  the  joystick  controller 


Making  joystick  work  right _ 


Maintaining  orientation 


Lazing 


como _ _ 

Communicate  (Team  &  Higher  HQ) 
Coordinate  with  team  member 


\ 


SECTION  II  (experiences) 

1,  For  each  item  in  the  table  below,  please  indicate  your  experience 
level  using  the  following  scale: 

1  =  none 

2  =  limited 

3  =  moderate 

4  =  considerable 


Eauipment/Simulated  Environments 

Experience  Level 

Global  Positioning  System  (GPS) 

Laser  Range  Finders 

Target  Designators 

Helmet  Mounted  Displays 

SIMNET-Tvpe  Environments 

JANUS 

Sending/Receiving  E-mail 

Laptop  Computers 

Thermal  Sights 

other  Simulations  (name  &  describe) 

2. 


Have  you  ever  navigated  in  a  terrain  data  base?  Yes 
jf  yes,  describe  where,  when,  mounted,  dismounted. 


_  No 

etc. )  . 


SECTION  III  (focusing) 

Answer  by  marking  an  "X"  in  the  appropriate  box  of  the  seven  point 
scale.  Consider  the  whole  scale  when  responding,  as  intermediate  levels 
may  apply.  For  example,  if  your  response  is  once  or  twice,  mark  the  2nd 
box  from  the  left.  If  your  response  is  many  times  but  not  extremely 
often,  then  the  6th  (or  2nd  box  from  the  right)  should  be  marked. 

1.  Do  you  ever  become  so  involved  in  a  video  game  that  it  is  as  if  you 
are  inside  the  game  rather  than  moving  a  joystick  and  watching  the 
screen? 

Ill  I _ I _ I _ I - 

never  occasionally  often 

2.  How  good  are  you  at  blocking  out  external  distractions  when  you  are 
involved  in  something? 


NOT  VERY 
GOOD 


SOMEWHAT 

GOOD 


VERY  GOOD 
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Learning  to  Use  the  Equipment 

Please  use  the  following  scale.  Indicate  your  answer  in  the  table 
How  difficult  was  it  to  learn  to  use  the  equipment  or  perform  the  task? 

1  Very  Easy 

2  Easy 

3  Neither  Easy  nor  Difficult 

4  Difficult 

5  Very  Difficult 


Eauipment  Name 

Difficulty 

Dial  a  Tank 

Drive  the  vehicle 

Maintain  orientation 

Naviaate  in  virtual  environment 

Treadport 

Chanae  orientation 

Keep  your  balance 

Navigate 

Map  of  the  Future 

Maintain  map  orientation 

Keep  track  of  things  on  map 

Learn  the  menus 

Respond  to  a  message 

Call  for  fire/send  message 

Grunt  II  PDA 

Understand  menu 

Realize  a  message  is  waiting  I 

Remember  PDA  functions 

Compose  text  messages 

Make  PDA  do  what  you  wanted 

Adi us t  fires  with  PDA 

GATH/Melios 

Detect  targets 

Identify  targets  (name  them) 

Designate  targets  (laze) 

Perform  BDA 

COVER 

Detect  targets 

Identify  targets  (name  them) 

Designate  targets  (laze) 

Perform  BDA 

Use  the  joystick  controller 

Maintain  orientation 
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Simulator  Questionnaire 


Section  1  SIMULATOR  REALISM 

Characterize  your  experience  in  the  simulation,  by  marking  an  ''X'  in  the 
appropriate  box  of  the  7-point  scale.  Consider  the  entire  scale  when 
making  your  responses,  as  intermediate  levels  may  apply.  Answer  the 
questions  in  the  order  that  they  appear.  Do  not  skip  questions  or 
return  to  a  previous  question  to  change  your  answer. 


1.  How  natural  did  your  interactions  with  the  environment  seem? 


EXTREMELY 

ARTIFICIAL 


I _ I. 

BORDERLINE 


_ I 

COMPLETELY 

NATURAL 


2.  How  much  did  the  visual  aspects  of  the  environment  involve  you? 


NOT  AT  ALL 


SOMEWHAT 


COMPLETELY 


3.  How  natural  was  the  mechanism  which  controlled  movement  through  the 
environment? 


EXTREMELY  BORDERLINE 

ARTIFICIAL 


COMPLETELY 

NATURAL 


4.  How  compelling  was  your  sense  of  objects  moving  through  space? 

I  I  I  _l _ 1 _ I _ I - • 

NOT  AT  ALL  MODERATELY  VERY 

COMPELLING  COMPELLING 


5.  How  much  did  your  experiences  in  the  virtual  environment  seem 
consistent  with  your  real  world  experiences? 


not  MODERATELY  VERY 

CONSISTENT  CONSISTENT  CONSISTENT 


6.  Were  you  able  to  anticipate  what  would  happen  next  in  response  to 
the  actions  that  you  performed? 


NOT  AT  ALL 


SOMEWHAT 


COMPLETELY 
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7,  How  completely  were  you  able  to  actively  survey  or  search  the 
environment  using  vision? 


I _ I 

NOT  AT  ALL 


SOMEWHAT 


COMPLETELY 


8,  How  compelling  was  your  sense  of  moving  around  inside  the  virtual 
environment? 


not  moderately  very 

COMPELLING  COMPELLING  COMPELLING 

9.  How  closely  were  you  able  to  examine  objects? 


NOT  AT  ALL  PRETTY  VERY 

CLOSELY  CLOSELY 


10-  How  well  could  you  examine  objects  from  multiple  viewpoints? 

1  I  I  _ i__ _ I  I _ > - > 

NOT  AT  ALL  SOMEWHAT  EXTENSIVELY 


11.  How  involved  were  you  in  the  virtual  environment  experience? 


I _ 

NOT 

INVOLVED 


MILDLY 

INVOLVED 


COMPLETELY 

ENGROSSED 


12 .  How  much  delay  did  you  experience  between  your  actions  and  expected 
outcomes? 


NO  DELAYS  MODERATE  LONG 

DELAYS  DELAYS 


13.  How  quickly  did  you  adjust  to  the  virtual  environment  experience? 


NOT  AT  ALL  SLOWLY  LESS  THAN 

ONE  MINUTE 

14.  How  proficient  in  moving  and  interacting  with  the  virtual 
environment  did  you  feel  at  the  end  of  the  experience? 


not  reasonably  very 

PROFICIENT  PROFICIENT  PROFICIENT 
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15.  Were  you  involved  in  the  scenarios 
track  of  time? 


to  the  extent  that  you  lost 


NOT  AT  ALL 


SOMEWHAT 


_ I 

COMPLETELY 


16.  Were  there  moments  during  the  scenarios  when  you  felt  completely 
focused  on  the  task  or  environment? 


NONE 


OCCASIONALLY 


FREQUENTLY 


17.  How  easily  did  you  adjust  to  the  control  devices  used  to  interact 
with  the  virtual  environment? 


_ I. 

DIFFICULT 


MODERATE 


_ 1 

EASILY 


Section  2  -  DISTRACTORS 


1.  How  much  did  the  visual  display  quality  interfere  or  distract  you 
from  performing  assigned  tasks  or  required  activities? 


I _ I. 

NOT  AT  ALL 


I _ I _ I _ I _ ' 

INTERFERED  PREVENTED 

SOMEWHAT  TASK  PERFORMANCE 


2.  How  well  could  you  concentrate  on  the  assigned  tasks  or  required 
activities  rather  than  on  the  mechanisms  used  to  perform  those  tasks  or 
activities? 


NOT  AT  ALL 


SOMEWHAT 


I _ I 

COMPLETELY 


3.  To  what  extent  did  events  occurring  outside  the  virtual  environment 
distract  from  your  experience  in  the  virtual  environment? 


NOT  AT  ALL 


MODERATELY 


_ I 

VERY  MUCH 


4.  Overall,  how  much  did  you  focus  on  using  the  display  and  control 
devices  instead  of  the  virtual  experience  and  experimental  tasks? 


1 _ 1. 

NOT  AT  ALL 


SOMEWHAT 


VERY  MUCH 
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Results  of  the  Participant  Immersive  Tendencies 

Questionnaire 


1.  Do  you  ever  become  so  involved  in  a  video  game  that  it  is  as 
if  you  are  inside  the  game  rather  than  moving  a  joystick  and 
watching  the  screen? 


1  XX 

1  X 

1  1  X  1  1 

1  XX  1  1 

NEVER 

OCCASIONALLY 

OFTEN 

2.  How 

good  are 

you  at  blocking  out  external 

distractions  when 

you  are 

involved 

in  something? 

1 

1  X 

1  1  1  XX  1 

XXX  1  1 

NOT  VERY 

SOMEWHAT 

VERY  GOOD 

GOOD 

GOOD 

3.  When  watching  sports,  do  you  ever  become  so  involved  in  the 
game  that  you  react  as  if  you  were  one  of  the  players? 


X  1 

_ ^XXX _ |_ 

X 

1  X  1  1 

1 

NEVER 

OCCASIONALLY 

OFTEN 

4.  When  playing  sports,  do  you  become  so  involved  in  the  game 
that  you  lose  track  of  time? 


X  1 

XX  1 

1  X  1 

1  X 

1  X 

NEVER 

OCCASIONALLY 

OFTEN 

5.  How  well  do  you  concentrate  on  enjoyable  activities? 


NOT  AT  ALL 


1  X  I _ 1 _ ^XXX _ I _ ^XX _ I 

MODERATELY  .  VERY  WELL 

WELL 


6.  How  often  do  you  play  arcade  or  video  games?  (OFTEN  should 
be  taken  to  mean  every  day  of  every  two  days,  on  average,) 

1  X  I  XXX  I  X  I  IX  1 _ I _ 

NEVER  OCCASIONALLY  OFTEN 

7 .  Do  you  ever  become  so  involved  in  doing  something  that  you 
lose  all  track  of  time? 

I _ I _ ^XXX _ I _ ^X _ I _ ^X _ I  X  I _ I 

NEVER  OCCASIONALLY  OFTEN 


8 .  How  easily  can  you  switch  attention  from  the  activity  in 
which  you  are  currently  involved  to  a  new  and  completely 
different  activity? 

X  1  X  I  X  I  XXX  _ 1 

FAIRLY  QUITE 

EASILY  EASILY 


NOT  SO 
EASILY 


9.  How  often  do  you  play  games  on  computers? 

I  X  1 _ ^XX _ I _ I _ 1 _ _ I 

NOT  AT  ALL  OCCASIONALLY 


FREQUENTLY 


10.  How  many  different  video,  computer,  or  arcade  games  have 
you  become  reasonably  good  at  playing? 


I  X  I _ I _ ^XX _ I  XI  X__ 

NONE  ONE  TWO  THREE  FOUR 

MORE 


_ I  X  I 

FIVE  SIX  OR 


11.  Have  you  ever  felt  completely  caught  up  in  an  experience, 
aware  of  everything  going  on  and  completely  open  to  all  of  it? 

1  1  XXX  I  1 _ I _ ^XX _ I _ ^X _ I _ ^1 

NEVER  OCCASIONALLY  FREQUENTLY 


12.  Are  you  easily  disturbed  when  involved  in  an  activity  or 
working  on  a  task? 

I  I  XXX  1  XX _ I  X  I _ I _ I _ ^ 

NEVER  OCCASIONALLY  OFTEN 
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Immersion  Questionaire  Results 


Table  I-l 

Immersion  Question  Means  For  The  First  And  Second 
Administrations 


Stem 

Admin 

1 

Admin 

2 

Mean 

1.  How  natural  did  your  interactions 
with  the  environment  seem? 

3.00 

3.83 

3.42 

2 .  How  much  did  the  visual  aspects  of 
the  environment  involve  you? 

4.50 

4.17 

4.33 

3.  How  natural  was  the  mechanism 
which  controlled  movement  through  the 
environment? 

3.00 

3.83 

3.42 

4 .  How  compelling  was  your  sense  of 
objects  moving  through  space? 

3.67 

5.17 

4.42 

5 .  How  much  did  your  experiences  in 
the  virtual  environment  seem 
consistent  with  your  real  world 
experiences? 

2.83 

3.33 

3.08 

6.  Were  you  able  to  anticipate  what 
would  happen  next  in  response  to  the 
actions  that  you  performed? 

4.17  . 

5.00 

4.58 

7.  How  completely  were  you  able  to 
actively  survey  or  search  the 
environment  using  vision? 

3.83 

4.00 

3.92 

8 .  How  compelling  was  your  sense  of 
moving  around  inside  the  virtual 
environment? 

4.00 

4.00 

4.00 

9.  How  closely  were  you  able  to 
examine  objects? 

3.33 

4.33 

3.83 

10.  How  well  could  you  examine 
objects  from  multiple  viewpoints? 

3.00 

3.67 

3.33 

11.  How  involved  were  you  in  the 
virtual  environment  experience? 

4.83 

4.50 

4.67 

12.  How  much  delay  did  you  experience 
between  your  actions  and  expected 
outcomes? 

3.33 

3.50 

3.42 

13.  How  quickly  did  you  adjust  to  the 
virtual  environment  experience? 

5.67 

5.67 

5.67 

14.  How  proficient  in  moving  and 
interacting  with  the  virtual 
environment  did  you  feel  at  the  end  of 
the  experience? 

4.83 

5.00 

4.92 

15.  Were  you  involved  in  the  3.17  4.00  3... 

scenarios  to  the  extent  that  you  lost 

track  of  time? _ _ _ _ _ _ _ 

16.  Were  there  moments  during  the  4.33  4.67  4.5 

scenarios  when  you  felt  completely 

focused  on  the  task  or  environment? _ 

17 .  How  easily  did  you  adjust  to  the  4.67  4.50  4.5 

control  devices  used  to  interact  with 

the  virtual  environment? _ _ _ 

D1 .  How  much  did  the  visual  display  3.50  4.17  3.E 

quality  interfere  or  distract  you  from 
performing  assigned  tasks  or  required 

activities? _ _ _ 

D2 .  How  well  could  you  concentrate  on  4.67  4.50  4.f 

the  assigned  tasks  or  required 
activities  rather  than  on  the 
mechanisms  used  to  perform  those  tasks 

or  activities? _ _ _ 

D3.  To  what  extent  did  events  4.17  4.50  4.1 

occurring  outside  the  virtual 
environment  distract  from  your 

experience  in  the  virtual  environment? _ 

D4 .  Overall,  how  much  did  you  focus  2.67  2.67  2.i 

on  using  the  display  and  control 
devices  instead  of  the  virtual 

experience  and  experimental  tasks? _ | _ | _ | _ 

Note .  Scale  values  range  from  1  to  7,  with  1  indicating  low 
immersion  and  7  indicating  high  immersion. 


Table  1-2 


Distribution  of  Immersion  Questionnaire  Responses 
(Administrations  1  And  2  Combined) 


1 .  How  natural  did  your  interactions  with  the  environment  seem? 


1  I  XXXX  _| _ XX _ I  XXX  I _ XXX _ 

EXTREMELY  BORDERLINE 

COMPLETELY 

ARTIFICIAL 


NATURAL 


2.  How  much  did  the  visual  aspects  of  the  environment  involve 
you? 


NOT  AT  ALL 


I _ XXXXX_ 

SOMEWHAT 


XXX 


XX 


COMPLETELY 


3.  How  natural  was  the  mechanism  which  controlled  movement 
through  the  environment? 


1 _ XX  I _ 1 _ XXXX _ I  XXX  I _ ^XXX 

EXTREMELY  BORDERLINE 

ARTIFICIAL 


COMPLETELY 

NATURAL 


4.  How  compelling  was  your  sense  of  objects  moving  through 
space? 


I _ l__X_ 

NOT  AT  ALL 


XXX _ I  XX  I _ ^XX _ 1  _XXXX _ 1 _ 1 

MODERATELY  VERY 

COMPELLING  COMPELLING 


5 .  How  much  did  your  experiences  in  the  virtual  environment 
seem  consistent  with  your  real  world  experiences? 


I  X  I _ ^XXX 

NOT 

CONSISTENT 


XXX _ 1  _XXXXX_  i _ ^X 

MODERATELY 

CONSISTENT 


VERY 

CONSISTENT 


6.  Were  you  able  to  anticipate  what  would  happen  next  in 
response  to  the  actions  that  you  performed? 

I  I  XX  1  I  XX  I ^XXXXX_| _ ^XXX 1 _ _l 

NOT  AT  ALL  SOMEWHAT  COMPLETELY 
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7 .  How  completely  were  you  able  to  actively  survey  or  search 
the  environment  using  vision? 


I _ I _ ^XX 

NOT  AT  ALL 


XX  I _ ^XXXX _ I _ ^XXX 

SOMEWHAT 


X _ 1 _ 

COMPLETELY 


8 .  How  compelling  was  your  sense  of  moving  around  inside  the 
virtual  environment? 


NOT 

COMPELLING 


XXXX 1  _XXXXX_  I _ ^XX _ 1 _ ^X _ I _ I 

MODEEIATELY  VERY 

COMPELLING  COMPELLING 


9.  How  closely  were  you 

1 _ I _ XXXX_  I _ XX 

NOT  AT  ALL 


able  to  examine  objects? 

_ 1  X _ I _ ^XX _ 1 _ ^XXX 

PRETTY 

CLOSELY 


VERY 

CLOSELY 


10.  How  well  could  you  examine  objects  from  multiple 
viewpoints? 


1  X  1  X  1  _XXX _ I  _XXXX_ 

NOT  AT  ALL  SOMEWHAT 


XX _ 1 _ ^X _ I _ 

EXTENSIVELY 


11.  How  involved  were  you  in  the  virtual  environment 
experience? 

1  I  I  X  I  XXXX  I  XXXXX _ I _ ^XX _ I _ I 

not  mildly  COMPLETELY 

INVOLVED  INVOLVED  ENGROSSED 

12 .  How  much  delay  did  you  experience  between  your  actions  and 
expected  outcomes? 


NO  DELAYS 


X  I  XXXXX_| _ ^XXXX _ I _ ^XX _ 1 _ 

MODERATE  LONG 

DELAYS  DELAYS 


13.  How  quickly  did  you  adjust  to  the  virtual  environment 
experience? 


NOT  AT  ALL 


X  I  XXX  1XXXXXXX_| _ ^X _ 1 

SLOWLY  LESS  THAN 

ONE  MINUTE 
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14.  How  proficient  in  moving  and  interacting  with  the  virtual 
environment  did  you  feel  at  the  end  of  the  experience? 

,111  XXXX _ 1_XXXXX _ 1 _ ^XXX _ 1 _ 

— not  reasonably  very 

PROFICIENT  PROFICIENT 

PROFICIENT 


15.  Were  you  involved  in  the  scenarios  to  the  extent  that  you 
lost  track  of  time? 


I _ ^X _ 1 _ ^XXX 

NOT  AT  ALL 


XX _ I  XX  I _ ^XX _ 

SOMEWHAT 


XX _ 1 _ I 

COMPLETELY 


16.  Were  there  moments  during  the  scenarios  when  you  felt 
completely  focused  on  the  task  or  environment? 


NONE 


XX  I  XXXX _ I _ ^XXXX _ 1 _ XX  I _ I 

OCCASIONALLY  FREQUENTLY 


17.  How  easily  did  you  adjust  to  the  control  devices  used  to 
interact  with  the  virtual  environment? 


DIFFICULT 


XXX _ I  XX  I _ XXXX _ 1 _ ^XXX _ I _ 

MODERATE  EASILY 


D1 .  How  much  did  the  visual  display  quality  interfere  or 
distract  you  from  performing  assigned  tasks  or  required 
activities? 


I  XX  1  XI  XXX _ 1_XXXXX_I _ ^X 

NOT  AT  ALL  INTERFERED 

SOMEWHAT 


I _ I 

PREVENTED 

TASK 

PERFORMANCE 


D2.  How  well  could  you  concentrate  on  the  assigned  tasks  or 
required  activities  rather  than  on  the  mechanisms  used  to 
perform  those  tasks  or  activities? 


NOT  AT  ALL 


XX  1  XXX _ I _ ^XXXXX_| _ ^XX _ 1 _ .1 

SOMEWHAT  COMPLETELY 


D3.  To  what  extent  did  events  occurring  outside  the  virtual 
environment  distract  from  your  experience  in  the  virtual 
environment? 

I  XXX  I  XXXXX  1  X _ 1 _ ^X _ 1 _ ^XX _ I _ I - 1 

NOT^AT  ALL  MODERATELY  VERY  MUCH 
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D4  .  Overall,  how  much  did  you  focus  on  using  the 
control  devices  instead  of  the  virtual  experience  an 
experimental  tasks? 


NOT  AT  ALL 


X  |_XXXXXXX1 _ ^XXX 

SOMEWHAT 


X _ 1 _ I 

VERY  MUCH 
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